Switch to new configuration system
Dear yt devs, TLDR: should we change the configuration format from .ini to another one (e.g. toml)? ------------------------------------------------------------------------ Last meeting we discussed what steps were required for the 4.0 release and one of the point was whether we should change how we handle configurations files (e.g. to get https://github.com/yt-project/yt/pull/1931 eventually in). Given that changing the format of yt's configuration is a big breaking change, this would have its place in a yt-3 to 4 transition plan. Amon the possibilities of a new or improved configuration system, users may be able to set some preferences per-field (e.g. have a configuration that implements the user's wish that "velocity_divergence should be displayed using a linear scale, with the seismic colormap"), or to pick their favourite unit system (e.g. cgs for the astro community, mks for the others). I know this has already been discussed earlier (both in PR comments and /via/ email), but let's take a moment to think about it with a fresh perspective. I can see three possible options: 1. Keep the .ini file format and build on top of that. _Pros:_ * no need to write a script to convert the old configuration files to a new format * easy to write for the user _Cons:_ * may become increasingly cumbersome to maintain if the available configuration keys are extended * no support for subgroups within one configuration group, so the configuration file may become cluttered or we'd need to handle this manually in the codebase 2. Switch to toml _Pros:_ * the language support semantically boolean/str/numbers (and parse them accordingly) * language is standardized & similar to .INI * easy to write (but see below) * possible to have nested configs, e.g. to set defaults for a range of different fields: [yt.fields.gas] # Use linear scale by default for gas fields … use_log = false [yt.fields.gas.pressure] # … except for pressure fields use_log = true That would be parsed natively as {'yt': { 'fields': { 'gas': { 'use_log': False, 'pressure':{ 'use_log': True } } } } * the syntax make it explicit what type each value is, so we could check the validity of the config when yt is loaded (for ex. [yt]\nloglevel=nan is invalid because loglevel should be a number). _Cons:_ * users will have to migrate their old config (if any), though we could provide a migration script * yet-another dependency (but we could vendor the parser, there are small implementations) * may be harder to write (how does a user know the expected type of each key? Should [yt]\n loglevel by a string or a number?) * for developers: we'd need to change the way the config is supported in the codebase (currently we're relying on ConfigParser) * there are some gotchas, for example boolean true is written "true" not "True" 3. Switch to yaml or json? What is your opinion? As a starting point, you could think about how you'd implement the example given above to deactivate the log scale for plots made with any field with type "gas", except pressure (is this something we want? If so, what's the best solution to achieve it in a meaningful way for the user?). Regards, Corentin -- Dr. Corentin Cadiou Post Doctoral Research Assistant Cosmoparticle Initiative Hub, desk 27 University College London (UCL) Gower St, Bloomsbury, London WC1E 6BT mobile: +33.6.43.18.66.83
Hi Corentin, This is a great idea. I was originally opposed to toml. But, I think it's now a transitive dependency for yt anyway, so we should be safe using it, and I think that it's reasonably common enough that it will be usable for folks. I would wager than migration is not going to be too terribly difficult, as we can probably parse a config, see if it matches the defaults, and if not, just bail and say that it should be converted. On Mon, Nov 23, 2020 at 8:40 AM Corentin CADIOU <corentin.cadiou@iap.fr> wrote:
Dear yt devs,
TLDR: should we change the configuration format from .ini to another one (e.g. toml)? ------------------------------
Last meeting we discussed what steps were required for the 4.0 release and one of the point was whether we should change how we handle configurations files (e.g. to get https://github.com/yt-project/yt/pull/1931 eventually in). Given that changing the format of yt's configuration is a big breaking change, this would have its place in a yt-3 to 4 transition plan.
Amon the possibilities of a new or improved configuration system, users may be able to set some preferences per-field (e.g. have a configuration that implements the user's wish that "velocity_divergence should be displayed using a linear scale, with the seismic colormap"), or to pick their favourite unit system (e.g. cgs for the astro community, mks for the others).
I know this has already been discussed earlier (both in PR comments and *via* email), but let's take a moment to think about it with a fresh perspective. I can see three possible options:
1. Keep the .ini file format and build on top of that. *Pros:* - no need to write a script to convert the old configuration files to a new format - easy to write for the user *Cons:* - may become increasingly cumbersome to maintain if the available configuration keys are extended - no support for subgroups within one configuration group, so the configuration file may become cluttered or we'd need to handle this manually in the codebase 2. Switch to toml *Pros:* - the language support semantically boolean/str/numbers (and parse them accordingly) - language is standardized & similar to .INI - easy to write (but see below) - possible to have nested configs, e.g. to set defaults for a range of different fields:
[yt.fields.gas] # Use linear scale by default for gas fields … use_log = false
[yt.fields.gas.pressure] # … except for pressure fields use_log = true
That would be parsed natively as {'yt': { 'fields': { 'gas': { 'use_log': False, 'pressure': { 'use_log': True } } } } - the syntax make it explicit what type each value is, so we could check the validity of the config when yt is loaded (for ex. [yt]\nloglevel=nan is invalid because loglevel should be a number). *Cons:* - users will have to migrate their old config (if any), though we could provide a migration script - yet-another dependency (but we could vendor the parser, there are small implementations) - may be harder to write (how does a user know the expected type of each key? Should [yt]\n loglevel by a string or a number?) - for developers: we'd need to change the way the config is supported in the codebase (currently we're relying on ConfigParser) - there are some gotchas, for example boolean true is written "true" not "True" 3. Switch to yaml or json?
What is your opinion? As a starting point, you could think about how you'd implement the example given above to deactivate the log scale for plots made with any field with type "gas", except pressure (is this something we want? If so, what's the best solution to achieve it in a meaningful way for the user?).
Regards, Corentin
-- Dr. Corentin Cadiou Post Doctoral Research Assistant Cosmoparticle Initiative Hub, desk 27 University College London (UCL) Gower St, Bloomsbury, London WC1E 6BT
mobile: +33.6.43.18.66.83
_______________________________________________ yt-dev mailing list -- yt-dev@python.org To unsubscribe send an email to yt-dev-leave@python.org https://mail.python.org/mailman3/lists/yt-dev.python.org/ Member address: matthewturk@gmail.com
Hi, I support changing the format to toml, which feels infinitely less controversial than yaml or json since it’s already so close to ini. Correct me if I’m wrong, but that would allow for the config to be accessed internally as a dictionary instead of through a ConfigParser instance, right ? I also support Matt’s suggestion for migration strategy, provided we don’t expect silent parsing errors to occur (e.g. True being parsed as a string instead of a boolean. This example is not relevant but you get the idea). Cheers, Clément
On 23 Nov 2020, at 16:05, Matthew Turk <matthewturk@gmail.com> wrote:
Hi Corentin,
This is a great idea.
I was originally opposed to toml. But, I think it's now a transitive dependency for yt anyway, so we should be safe using it, and I think that it's reasonably common enough that it will be usable for folks. I would wager than migration is not going to be too terribly difficult, as we can probably parse a config, see if it matches the defaults, and if not, just bail and say that it should be converted.
On Mon, Nov 23, 2020 at 8:40 AM Corentin CADIOU <corentin.cadiou@iap.fr> wrote:
Dear yt devs,
TLDR: should we change the configuration format from .ini to another one (e.g. toml)?
---------------------------------------------------------------
Last meeting we discussed what steps were required for the 4.0 release and one of the point was whether we should change how we handle configurations files (e.g. to get https://github.com/yt-project/yt/pull/1931 eventually in). Given that changing the format of yt's configuration is a big breaking change, this would have its place in a yt-3 to 4 transition plan.
Amon the possibilities of a new or improved configuration system, users may be able to set some preferences per-field (e.g. have a configuration that implements the user's wish that "velocity_divergence should be displayed using a linear scale, with the seismic colormap"), or to pick their favourite unit system (e.g. cgs for the astro community, mks for the others).
I know this has already been discussed earlier (both in PR comments and via email), but let's take a moment to think about it with a fresh perspective. I can see three possible options:
- Keep the .ini file format and build on top of that. Pros:
- no need to write a script to convert the old configuration files to a new format - easy to write for the user Cons:
- may become increasingly cumbersome to maintain if the available configuration keys are extended - no support for subgroups within one configuration group, so the configuration file may become cluttered or we'd need to handle this manually in the codebase - Switch to toml Pros:
- the language support semantically boolean/str/numbers (and parse them accordingly) - language is standardized & similar to .INI - easy to write (but see below) - possible to have nested configs, e.g. to set defaults for a range of different fields:
[yt.fields.gas] # Use linear scale by default for gas fields … use_log = false
[yt.fields.gas.pressure] # … except for pressure fields use_log = true
That would be parsed natively as {'yt': { 'fields': {'gas': {'use_log': False,'pressure':{ 'use_log': True }}} } - the syntax make it explicit what type each value is, so we could check the validity of the config when yt is loaded (for ex. [yt]\nloglevel=nan is invalid because loglevel should be a number). Cons:
- users will have to migrate their old config (if any), though we could provide a migration script - yet-another dependency (but we could vendor the parser, there are small implementations) - may be harder to write (how does a user know the expected type of each key? Should [yt]\n loglevel by a string or a number?) - for developers: we'd need to change the way the config is supported in the codebase (currently we're relying on ConfigParser) - there are some gotchas, for example boolean true is written "true" not "True"
- Switch to yaml or json?
What is your opinion? As a starting point, you could think about how you'd implement the example given above to deactivate the log scale for plots made with any field with type "gas", except pressure (is this something we want? If so, what's the best solution to achieve it in a meaningful way for the user?).
Regards, Corentin
-- Dr. Corentin Cadiou Post Doctoral Research Assistant Cosmoparticle Initiative Hub, desk 27 University College London (UCL) Gower St, Bloomsbury, London WC1E 6BT
mobile: +33.6.43.18.66.83
_______________________________________________ yt-dev mailing list -- yt-dev@python.org To unsubscribe send an email to yt-dev-leave@python.org https://mail.python.org/mailman3/lists/yt-dev.python.org/ Member address: matthewturk@gmail.com
Hi there, I support the idea of providing a conversion script and printing out a command to run to do the conversion. This needs to be a disruption that takes people less than 60 second to return to their workflow. Britton On Mon, Nov 23, 2020 at 8:53 PM Clément Robert via yt-dev <yt-dev@python.org> wrote:
Hi, I support changing the format to toml, which feels infinitely less controversial than yaml or json since it’s already so close to ini. Correct me if I’m wrong, but that would allow for the config to be accessed internally as a dictionary instead of through a ConfigParser instance, right ?
I also support Matt’s suggestion for migration strategy, provided we don’t expect silent parsing errors to occur (e.g. True being parsed as a string instead of a boolean. This example is not relevant but you get the idea).
Cheers, Clément
On 23 Nov 2020, at 16:05, Matthew Turk <matthewturk@gmail.com> wrote:
Hi Corentin,
This is a great idea.
I was originally opposed to toml. But, I think it's now a transitive dependency for yt anyway, so we should be safe using it, and I think that it's reasonably common enough that it will be usable for folks. I would wager than migration is not going to be too terribly difficult, as we can probably parse a config, see if it matches the defaults, and if not, just bail and say that it should be converted.
On Mon, Nov 23, 2020 at 8:40 AM Corentin CADIOU <corentin.cadiou@iap.fr> wrote:
Dear yt devs,
TLDR: should we change the configuration format from .ini to another one (e.g. toml)? ------------------------------
Last meeting we discussed what steps were required for the 4.0 release and one of the point was whether we should change how we handle configurations files (e.g. to get https://github.com/yt-project/yt/pull/1931 eventually in). Given that changing the format of yt's configuration is a big breaking change, this would have its place in a yt-3 to 4 transition plan.
Amon the possibilities of a new or improved configuration system, users may be able to set some preferences per-field (e.g. have a configuration that implements the user's wish that "velocity_divergence should be displayed using a linear scale, with the seismic colormap"), or to pick their favourite unit system (e.g. cgs for the astro community, mks for the others).
I know this has already been discussed earlier (both in PR comments and *via* email), but let's take a moment to think about it with a fresh perspective. I can see three possible options:
1. Keep the .ini file format and build on top of that. *Pros:* - no need to write a script to convert the old configuration files to a new format - easy to write for the user *Cons:* - may become increasingly cumbersome to maintain if the available configuration keys are extended - no support for subgroups within one configuration group, so the configuration file may become cluttered or we'd need to handle this manually in the codebase 2. Switch to toml *Pros:* - the language support semantically boolean/str/numbers (and parse them accordingly) - language is standardized & similar to .INI - easy to write (but see below) - possible to have nested configs, e.g. to set defaults for a range of different fields:
[yt.fields.gas] # Use linear scale by default for gas fields … use_log = false
[yt.fields.gas.pressure] # … except for pressure fields use_log = true
That would be parsed natively as {'yt': { 'fields': { 'gas': { 'use_log': False, 'pressure': { 'use_log': True } } } } - the syntax make it explicit what type each value is, so we could check the validity of the config when yt is loaded (for ex. [yt]\nloglevel=nan is invalid because loglevel should be a number). *Cons:* - users will have to migrate their old config (if any), though we could provide a migration script - yet-another dependency (but we could vendor the parser, there are small implementations) - may be harder to write (how does a user know the expected type of each key? Should [yt]\n loglevel by a string or a number?) - for developers: we'd need to change the way the config is supported in the codebase (currently we're relying on ConfigParser) - there are some gotchas, for example boolean true is written "true" not "True" 3. Switch to yaml or json?
What is your opinion? As a starting point, you could think about how you'd implement the example given above to deactivate the log scale for plots made with any field with type "gas", except pressure (is this something we want? If so, what's the best solution to achieve it in a meaningful way for the user?).
Regards, Corentin
-- Dr. Corentin Cadiou Post Doctoral Research Assistant Cosmoparticle Initiative Hub, desk 27 University College London (UCL) Gower St, Bloomsbury, London WC1E 6BT
mobile: +33.6.43.18.66.83
_______________________________________________ yt-dev mailing list -- yt-dev@python.org To unsubscribe send an email to yt-dev-leave@python.org https://mail.python.org/mailman3/lists/yt-dev.python.org/ Member address: matthewturk@gmail.com
_______________________________________________ yt-dev mailing list -- yt-dev@python.org To unsubscribe send an email to yt-dev-leave@python.org https://mail.python.org/mailman3/lists/yt-dev.python.org/ Member address: brittonsmith@gmail.com
Dear all, Thanks for the constructive feedback. I have hopefully taken all your comments into account and moved our configuration to toml. The pull request is here https://github.com/yt-project/yt/pull/2981. Following Britton's comment, I tried to keep the migration as simple as possible. It can be achieved by typing `yt config migrate` once. Note that toml is case-sensitive, while ConfigParser is not. To accommodate for this, the migration script will convert the case of known variables (ex. [yt] LOGLEVEL= 10 will be converted to [yt] loglevel = 10). Now, here are the novelties: * the format is now toml. * the configuration is loaded from ~/.config/yt/yt.toml (instead of ~/config/yt/ytrc), and if a local yt.toml file is found in the current folder or any of its parent, it is also loaded (and overrides previous preferences). You can use `yt config print-path --local` to ask yt where it found the local configuration file. * the local configuration file can be configured from the CLI in the same way as the global one, provided you add the `--local` flag, e.g. `yt config set --local yt loglevel 1` will set the loglevel to 1 in the local config and leave the global config unchanged. If no local config is found, a new one is created in the working directory. * the configuration is now type-sensitive (a useful message is thrown at the user if the type changes). See for example > import yt > yt.config.ytcfg["foo", "bar", "baz"] = 10 > yt.config.ytcfg["foo", "bar", "baz"] = "I cannot pass a string now" [...] TypeError: Error when setting foo.bar.baz. Tried to assign a value of type <class 'str'>, expected type <class 'int'>. This entry was last modified in runtime. > yt.config.ytcfg["yt", "loglevel"] = "invalid loglevel" [...] TypeError: Error when setting yt.loglevel. Tried to assign a value of type <class 'str'>, expected type <class 'int'>. This entry was last modified in file: /path/to/yt/installation/yt/yt.toml. This allows errors to be caught as soon as possible instead of the issue popping randomly when the actual configuration entry is used for the first time. * I added the possibility to find the most specific entry matching a given path. Once again, let me illustrate that with an example: imagine you have this configuration [fields] cmap = "arbre" [fields.gas] cmap = "viridis" [fields.gas.temperature] cmap = "plasma" then you can use `ytcfg.get_most_specific` to find the most specific option corresponding to your request. For example > yt.config.ytcfg.get_most_specific("fields", "gas", "temperature", "cmap") # full match with fields.gas.temperature:cmap 'plasma' > yt.config.ytcfg.get_most_specific("fields", "gas", "density", "cmap") # partial match with fields.gas:cmap 'viridis' > yt.config.ytcfg.get_most_specific("fields", "io", "particle_velocity", "cmap") # partial match with fields:cmap 'arbre' This enable us to set options for example for all fields, for all fields of type `gas` and for specific fields individually. The PR is ready to be merged, but I am very much open to suggestions of things to add or remove before it gets in. Cheers, Corentin On 24/11/2020 10:01, Britton Smith wrote:
Hi there,
I support the idea of providing a conversion script and printing out a command to run to do the conversion. This needs to be a disruption that takes people less than 60 second to return to their workflow.
Britton
On Mon, Nov 23, 2020 at 8:53 PM Clément Robert via yt-dev <yt-dev@python.org <mailto:yt-dev@python.org>> wrote:
Hi, I support changing the format to toml, which feels infinitely less controversial than yaml or json since it’s already so close to ini. Correct me if I’m wrong, but that would allow for the config to be accessed internally as a dictionary instead of through a ConfigParser instance, right ?
I also support Matt’s suggestion for migration strategy, provided we don’t expect silent parsing errors to occur (e.g. True being parsed as a string instead of a boolean. This example is not relevant but you get the idea).
Cheers, Clément
On 23 Nov 2020, at 16:05, Matthew Turk <matthewturk@gmail.com <mailto:matthewturk@gmail.com>> wrote:
Hi Corentin,
This is a great idea.
I was originally opposed to toml. But, I think it's now a transitive dependency for yt anyway, so we should be safe using it, and I think that it's reasonably common enough that it will be usable for folks. I would wager than migration is not going to be too terribly difficult, as we can probably parse a config, see if it matches the defaults, and if not, just bail and say that it should be converted.
On Mon, Nov 23, 2020 at 8:40 AM Corentin CADIOU <corentin.cadiou@iap.fr <mailto:corentin.cadiou@iap.fr>> wrote:
Dear yt devs,
TLDR: should we change the configuration format from .ini to another one (e.g. toml)?
------------------------------------------------------------------------
Last meeting we discussed what steps were required for the 4.0 release and one of the point was whether we should change how we handle configurations files (e.g. to get https://github.com/yt-project/yt/pull/1931 <https://github.com/yt-project/yt/pull/1931> eventually in). Given that changing the format of yt's configuration is a big breaking change, this would have its place in a yt-3 to 4 transition plan.
Amon the possibilities of a new or improved configuration system, users may be able to set some preferences per-field (e.g. have a configuration that implements the user's wish that "velocity_divergence should be displayed using a linear scale, with the seismic colormap"), or to pick their favourite unit system (e.g. cgs for the astro community, mks for the others).
I know this has already been discussed earlier (both in PR comments and /via/ email), but let's take a moment to think about it with a fresh perspective. I can see three possible options:
1. Keep the .ini file format and build on top of that. _Pros:_ * no need to write a script to convert the old configuration files to a new format * easy to write for the user _Cons:_ * may become increasingly cumbersome to maintain if the available configuration keys are extended * no support for subgroups within one configuration group, so the configuration file may become cluttered or we'd need to handle this manually in the codebase 2. Switch to toml _Pros:_ * the language support semantically boolean/str/numbers (and parse them accordingly) * language is standardized & similar to .INI * easy to write (but see below) * possible to have nested configs, e.g. to set defaults for a range of different fields:
[yt.fields.gas] # Use linear scale by default for gas fields … use_log = false
[yt.fields.gas.pressure] # … except for pressure fields use_log = true
That would be parsed natively as {'yt': { 'fields': { 'gas': { 'use_log': False, 'pressure':{ 'use_log': True } } } } * the syntax make it explicit what type each value is, so we could check the validity of the config when yt is loaded (for ex. [yt]\nloglevel=nan is invalid because loglevel should be a number). _Cons:_ * users will have to migrate their old config (if any), though we could provide a migration script * yet-another dependency (but we could vendor the parser, there are small implementations) * may be harder to write (how does a user know the expected type of each key? Should [yt]\n loglevel by a string or a number?) * for developers: we'd need to change the way the config is supported in the codebase (currently we're relying on ConfigParser) * there are some gotchas, for example boolean true is written "true" not "True" 3. Switch to yaml or json?
What is your opinion? As a starting point, you could think about how you'd implement the example given above to deactivate the log scale for plots made with any field with type "gas", except pressure (is this something we want? If so, what's the best solution to achieve it in a meaningful way for the user?).
Regards, Corentin
-- Dr. Corentin Cadiou Post Doctoral Research Assistant Cosmoparticle Initiative Hub, desk 27 University College London (UCL) Gower St, Bloomsbury, London WC1E 6BT
mobile: +33.6.43.18.66.83
_______________________________________________ yt-dev mailing list -- yt-dev@python.org <mailto:yt-dev@python.org> To unsubscribe send an email to yt-dev-leave@python.org <mailto:yt-dev-leave@python.org> https://mail.python.org/mailman3/lists/yt-dev.python.org/ <https://mail.python.org/mailman3/lists/yt-dev.python.org/> Member address: matthewturk@gmail.com <mailto:matthewturk@gmail.com>
_______________________________________________ yt-dev mailing list -- yt-dev@python.org <mailto:yt-dev@python.org> To unsubscribe send an email to yt-dev-leave@python.org <mailto:yt-dev-leave@python.org> https://mail.python.org/mailman3/lists/yt-dev.python.org/ <https://mail.python.org/mailman3/lists/yt-dev.python.org/> Member address: brittonsmith@gmail.com <mailto:brittonsmith@gmail.com>
_______________________________________________ yt-dev mailing list -- yt-dev@python.org To unsubscribe send an email to yt-dev-leave@python.org https://mail.python.org/mailman3/lists/yt-dev.python.org/ Member address: corentin.cadiou@iap.fr
-- Dr. Corentin Cadiou Post Doctoral Research Assistant Cosmoparticle Initiative Hub, desk 27 University College London (UCL) Gower St, Bloomsbury, London WC1E 6BT mobile: +33.6.43.18.66.83
Hi Corentin, thanks a lot for getting so much done in that direction so quickly. Regarding the `get_most_specific` feature, I’m wondering how flexible it is. For instance, can I specify a cmap for all “temperature” fields, independently from the ftype ? I imagine it would read something like [fields.temperature] cmap = “plasma” Another point I’d like to raise, and it’s not mentioned in your email (but we discussed it in private), is the behaviour in case more than one config file is found (in cwd and the global one for instance). So currently, according to our discussion, both config files would be taken into account, the local one taking priority in case of overlapping. I can imagine that one would like to build nested configurations from nested directories in a project (although I _hope_ no-one actually thinks this a good idea !), but what I’m worried about is the reproducibility among participants in a shared project. I wrote this issue a few month back https://github.com/yt-project/yt/issues/2635 Basically, allowing local config files enable them to be shared along with a notebook or script in a repo, but if the global config file is still having an effect, then the results may vary from one machine to the other and I don’t think that’s a desirable trait in general. On the other hand, it would be nice that the possibility remains available, but I think it should only be used if and when this behaviour is explicitly asked for. In other words, this could be a parameter defined in the local-most file ! Clément
On 26 Nov 2020, at 16:03, Corentin CADIOU <corentin.cadiou@iap.fr> wrote:
Dear all,
Thanks for the constructive feedback. I have hopefully taken all your comments into account and moved our configuration to toml. The pull request is here https://github.com/yt-project/yt/pull/2981.
Following Britton's comment, I tried to keep the migration as simple as possible. It can be achieved by typing `yt config migrate` once. Note that toml is case-sensitive, while ConfigParser is not. To accommodate for this, the migration script will convert the case of known variables (ex. [yt] LOGLEVEL= 10 will be converted to [yt] loglevel = 10).
Now, here are the novelties:
- the format is now toml. - the configuration is loaded from ~/.config/yt/yt.toml (instead of ~/config/yt/ytrc), and if a local yt.toml file is found in the current folder or any of its parent, it is also loaded (and overrides previous preferences). You can use `yt config print-path --local` to ask yt where it found the local configuration file. - the local configuration file can be configured from the CLI in the same way as the global one, provided you add the `--local` flag, e.g. `yt config set --local yt loglevel 1` will set the loglevel to 1 in the local config and leave the global config unchanged. If no local config is found, a new one is created in the working directory. - the configuration is now type-sensitive (a useful message is thrown at the user if the type changes). See for example
import yt yt.config.ytcfg["foo", "bar", "baz"] = 10 yt.config.ytcfg["foo", "bar", "baz"] = "I cannot pass a string now" [...] TypeError: Error when setting foo.bar.baz. Tried to assign a value of type <class 'str'>, expected type <class 'int'>. This entry was last modified in runtime. yt.config.ytcfg["yt", "loglevel"] = "invalid loglevel" [...] TypeError: Error when setting yt.loglevel. Tried to assign a value of type <class 'str'>, expected type <class 'int'>. This entry was last modified in file: /path/to/yt/installation/yt/yt.toml. This allows errors to be caught as soon as possible instead of the issue popping randomly when the actual configuration entry is used for the first time.
- I added the possibility to find the most specific entry matching a given path. Once again, let me illustrate that with an example: imagine you have this configuration
[fields] cmap = "arbre"
[fields.gas] cmap = "viridis"
[fields.gas.temperature] cmap = "plasma"
then you can use `ytcfg.get_most_specific` to find the most specific option corresponding to your request. For example
yt.config.ytcfg.get_most_specific("fields", "gas", "temperature", "cmap") # full match with fields.gas.temperature:cmap 'plasma' yt.config.ytcfg.get_most_specific("fields", "gas", "density", "cmap") # partial match with fields.gas:cmap 'viridis' yt.config.ytcfg.get_most_specific("fields", "io", "particle_velocity", "cmap") # partial match with fields:cmap 'arbre' This enable us to set options for example for all fields, for all fields of type `gas` and for specific fields individually.
The PR is ready to be merged, but I am very much open to suggestions of things to add or remove before it gets in.
Cheers, Corentin
On 24/11/2020 10:01, Britton Smith wrote:
Hi there,
I support the idea of providing a conversion script and printing out a command to run to do the conversion. This needs to be a disruption that takes people less than 60 second to return to their workflow.
Britton
On Mon, Nov 23, 2020 at 8:53 PM Clément Robert via yt-dev <yt-dev@python.org> wrote:
Hi, I support changing the format to toml, which feels infinitely less controversial than yaml or json since it’s already so close to ini. Correct me if I’m wrong, but that would allow for the config to be accessed internally as a dictionary instead of through a ConfigParser instance, right ?
I also support Matt’s suggestion for migration strategy, provided we don’t expect silent parsing errors to occur (e.g. True being parsed as a string instead of a boolean. This example is not relevant but you get the idea).
Cheers, Clément
On 23 Nov 2020, at 16:05, Matthew Turk <matthewturk@gmail.com> wrote:
Hi Corentin,
This is a great idea.
I was originally opposed to toml. But, I think it's now a transitive dependency for yt anyway, so we should be safe using it, and I think that it's reasonably common enough that it will be usable for folks. I would wager than migration is not going to be too terribly difficult, as we can probably parse a config, see if it matches the defaults, and if not, just bail and say that it should be converted.
On Mon, Nov 23, 2020 at 8:40 AM Corentin CADIOU <corentin.cadiou@iap.fr> wrote:
Dear yt devs,
TLDR: should we change the configuration format from .ini to another one (e.g. toml)?
---------------------------------------------------------------
Last meeting we discussed what steps were required for the 4.0 release and one of the point was whether we should change how we handle configurations files (e.g. to get https://github.com/yt-project/yt/pull/1931 eventually in). Given that changing the format of yt's configuration is a big breaking change, this would have its place in a yt-3 to 4 transition plan.
Amon the possibilities of a new or improved configuration system, users may be able to set some preferences per-field (e.g. have a configuration that implements the user's wish that "velocity_divergence should be displayed using a linear scale, with the seismic colormap"), or to pick their favourite unit system (e.g. cgs for the astro community, mks for the others).
I know this has already been discussed earlier (both in PR comments and via email), but let's take a moment to think about it with a fresh perspective. I can see three possible options:
- Keep the .ini file format and build on top of that. Pros:
- no need to write a script to convert the old configuration files to a new format - easy to write for the user Cons:
- may become increasingly cumbersome to maintain if the available configuration keys are extended - no support for subgroups within one configuration group, so the configuration file may become cluttered or we'd need to handle this manually in the codebase - Switch to toml Pros:
- the language support semantically boolean/str/numbers (and parse them accordingly) - language is standardized & similar to .INI - easy to write (but see below) - possible to have nested configs, e.g. to set defaults for a range of different fields:
[yt.fields.gas] # Use linear scale by default for gas fields … use_log = false
[yt.fields.gas.pressure] # … except for pressure fields use_log = true
That would be parsed natively as {'yt': { 'fields': {'gas': {'use_log': False,'pressure':{ 'use_log': True }}} } - the syntax make it explicit what type each value is, so we could check the validity of the config when yt is loaded (for ex. [yt]\nloglevel=nan is invalid because loglevel should be a number). Cons:
- users will have to migrate their old config (if any), though we could provide a migration script - yet-another dependency (but we could vendor the parser, there are small implementations) - may be harder to write (how does a user know the expected type of each key? Should [yt]\n loglevel by a string or a number?) - for developers: we'd need to change the way the config is supported in the codebase (currently we're relying on ConfigParser) - there are some gotchas, for example boolean true is written "true" not "True"
- Switch to yaml or json?
What is your opinion? As a starting point, you could think about how you'd implement the example given above to deactivate the log scale for plots made with any field with type "gas", except pressure (is this something we want? If so, what's the best solution to achieve it in a meaningful way for the user?).
Regards, Corentin
-- Dr. Corentin Cadiou Post Doctoral Research Assistant Cosmoparticle Initiative Hub, desk 27 University College London (UCL) Gower St, Bloomsbury, London WC1E 6BT
mobile: +33.6.43.18.66.83
_______________________________________________ yt-dev mailing list -- yt-dev@python.org To unsubscribe send an email to yt-dev-leave@python.org https://mail.python.org/mailman3/lists/yt-dev.python.org/ Member address: matthewturk@gmail.com
_______________________________________________ yt-dev mailing list -- yt-dev@python.org To unsubscribe send an email to yt-dev-leave@python.org https://mail.python.org/mailman3/lists/yt-dev.python.org/ Member address: brittonsmith@gmail.com
_______________________________________________ yt-dev mailing list -- yt-dev@python.org To unsubscribe send an email to yt-dev-leave@python.org
https://mail.python.org/mailman3/lists/yt-dev.python.org/ Member address: corentin.cadiou@iap.fr
-- Dr. Corentin Cadiou Post Doctoral Research Assistant Cosmoparticle Initiative Hub, desk 27 University College London (UCL) Gower St, Bloomsbury, London WC1E 6BT
mobile: +33.6.43.18.66.83
<OpenPGP_signature.sig>
Hi Clement, On 26/11/2020 16:01, Clément Robert via yt-dev wrote:
Hi Corentin, thanks a lot for getting so much done in that direction so quickly. Regarding the `get_most_specific` feature, I’m wondering how flexible it is. For instance, can I specify a cmap for all “temperature” fields, independently from the ftype ? I imagine it would read something like
[fields.temperature] cmap = “plasma”
Currently, the `get_most_specific` feature finds the longest consecutive common path between the config and the requested config item, starting from the left only. In other words, if you have [fields.temperature] cmap = "plasma" it will return "plasma" if you call `ytcfg.get_most_specific("fields", "temperature", "cmap")` but will fail with `ytcfg.get_most_specific("fields", "gas", "temperature", "cmap")`. The reason is that otherwise we may end up with ambiguous results, i.e. what should `ytcfg.get_most_specific("fields", "gas", "temperature", "cmap")` return with the following config? [fields.temperature] cmap = "plasma" [fields.gas] cmap = "not_plasma" We could of course fix the ambiguity by setting priority rules (ex. in this case, we could decide a field name is more specific than a field type, so it should have priority). What do others think about this?
Another point I’d like to raise, and it’s not mentioned in your email (but we discussed it in private), is the behaviour in case more than one config file is found (in cwd and the global one for instance). So currently, according to our discussion, both config files would be taken into account, the local one taking priority in case of overlapping. I can imagine that one would like to build nested configurations from nested directories in a project (although I _hope_ no-one actually thinks this a good idea !), but what I’m worried about is the reproducibility among participants in a shared project. I wrote this issue a few month back https://github.com/yt-project/yt/issues/2635 <https://github.com/yt-project/yt/issues/2635> Basically, allowing local config files enable them to be shared along with a notebook or script in a repo, but if the global config file is still having an effect, then the results may vary from one machine to the other and I don’t think that’s a desirable trait in general. On the other hand, it would be nice that the possibility remains available, but I think it should only be used if and when this behaviour is explicitly asked for. In other words, this could be a parameter defined in the local-most file !
TLDR; to ease the process of others weighing in, we have three options (I'd actually be in favor of Clement's option): 1. do not support local config files, 2. Clement's proposition: load the local config file (if any), otherwise load the global config file, 3. Corentin's proposition: load both the local config file (if any) and the global config file + provide a way to dump the current config /via/ a CLI argument. *More details:* The way the PR is implemented /currently/ is that it loads the config from ~/.config/yt/yt.toml, then it walks up the folder tree until it finds a yt.toml file. At the first match (if any), it reads it, overrides whatever was defined earlier and stops walking the tree up (i.e. at most one local yt.toml file will ever be read, in the same spirit as git stopping at the first parent folder that contains a .git directory). If I am correct, your proposition is the following: if any local config file is found, then do not read the global config (~/.config/yt/yt.toml). If we were to adopt this proposition, which file should `yt config set foo bar` configure? Another option would be to read both (as is currently the case in my PR), but have the cli option `yt config list` dump the full config (either global + local, or defaults + global + local). Currently, `yt config list` shows only the global config and `yt config list --local` only the local one.
Clément
On 26 Nov 2020, at 16:03, Corentin CADIOU <corentin.cadiou@iap.fr <mailto:corentin.cadiou@iap.fr>> wrote:
Dear all,
Thanks for the constructive feedback. I have hopefully taken all your comments into account and moved our configuration to toml. The pull request is here https://github.com/yt-project/yt/pull/2981.
Following Britton's comment, I tried to keep the migration as simple as possible. It can be achieved by typing `yt config migrate` once. Note that toml is case-sensitive, while ConfigParser is not. To accommodate for this, the migration script will convert the case of known variables (ex. [yt] LOGLEVEL= 10 will be converted to [yt] loglevel = 10).
Now, here are the novelties:
* the format is now toml. * the configuration is loaded from ~/.config/yt/yt.toml (instead of ~/config/yt/ytrc), and if a local yt.toml file is found in the current folder or any of its parent, it is also loaded (and overrides previous preferences). You can use `yt config print-path --local` to ask yt where it found the local configuration file. * the local configuration file can be configured from the CLI in the same way as the global one, provided you add the `--local` flag, e.g. `yt config set --local yt loglevel 1` will set the loglevel to 1 in the local config and leave the global config unchanged. If no local config is found, a new one is created in the working directory. * the configuration is now type-sensitive (a useful message is thrown at the user if the type changes). See for example
> import yt > yt.config.ytcfg["foo", "bar", "baz"] = 10 > yt.config.ytcfg["foo", "bar", "baz"] = "I cannot pass a string now" [...] TypeError: Error when setting foo.bar.baz. Tried to assign a value of type <class 'str'>, expected type <class 'int'>. This entry was last modified in runtime. > yt.config.ytcfg["yt", "loglevel"] = "invalid loglevel" [...] TypeError: Error when setting yt.loglevel. Tried to assign a value of type <class 'str'>, expected type <class 'int'>. This entry was last modified in file: /path/to/yt/installation/yt/yt.toml.
This allows errors to be caught as soon as possible instead of the issue popping randomly when the actual configuration entry is used for the first time.
* I added the possibility to find the most specific entry matching a given path. Once again, let me illustrate that with an example: imagine you have this configuration
[fields] cmap = "arbre"
[fields.gas] cmap = "viridis"
[fields.gas.temperature] cmap = "plasma"
then you can use `ytcfg.get_most_specific` to find the most specific option corresponding to your request. For example
> yt.config.ytcfg.get_most_specific("fields", "gas", "temperature", "cmap") # full match with fields.gas.temperature:cmap 'plasma' > yt.config.ytcfg.get_most_specific("fields", "gas", "density", "cmap") # partial match with fields.gas:cmap 'viridis' > yt.config.ytcfg.get_most_specific("fields", "io", "particle_velocity", "cmap") # partial match with fields:cmap 'arbre'
This enable us to set options for example for all fields, for all fields of type `gas` and for specific fields individually.
The PR is ready to be merged, but I am very much open to suggestions of things to add or remove before it gets in.
Cheers, Corentin
On 24/11/2020 10:01, Britton Smith wrote:
Hi there,
I support the idea of providing a conversion script and printing out a command to run to do the conversion. This needs to be a disruption that takes people less than 60 second to return to their workflow.
Britton
On Mon, Nov 23, 2020 at 8:53 PM Clément Robert via yt-dev <yt-dev@python.org <mailto:yt-dev@python.org>> wrote:
Hi, I support changing the format to toml, which feels infinitely less controversial than yaml or json since it’s already so close to ini. Correct me if I’m wrong, but that would allow for the config to be accessed internally as a dictionary instead of through a ConfigParser instance, right ?
I also support Matt’s suggestion for migration strategy, provided we don’t expect silent parsing errors to occur (e.g. True being parsed as a string instead of a boolean. This example is not relevant but you get the idea).
Cheers, Clément
On 23 Nov 2020, at 16:05, Matthew Turk <matthewturk@gmail.com <mailto:matthewturk@gmail.com>> wrote:
Hi Corentin,
This is a great idea.
I was originally opposed to toml. But, I think it's now a transitive dependency for yt anyway, so we should be safe using it, and I think that it's reasonably common enough that it will be usable for folks. I would wager than migration is not going to be too terribly difficult, as we can probably parse a config, see if it matches the defaults, and if not, just bail and say that it should be converted.
On Mon, Nov 23, 2020 at 8:40 AM Corentin CADIOU <corentin.cadiou@iap.fr <mailto:corentin.cadiou@iap.fr>> wrote:
Dear yt devs,
TLDR: should we change the configuration format from .ini to another one (e.g. toml)?
------------------------------------------------------------------------
Last meeting we discussed what steps were required for the 4.0 release and one of the point was whether we should change how we handle configurations files (e.g. to get https://github.com/yt-project/yt/pull/1931 <https://github.com/yt-project/yt/pull/1931> eventually in). Given that changing the format of yt's configuration is a big breaking change, this would have its place in a yt-3 to 4 transition plan.
Amon the possibilities of a new or improved configuration system, users may be able to set some preferences per-field (e.g. have a configuration that implements the user's wish that "velocity_divergence should be displayed using a linear scale, with the seismic colormap"), or to pick their favourite unit system (e.g. cgs for the astro community, mks for the others).
I know this has already been discussed earlier (both in PR comments and /via/ email), but let's take a moment to think about it with a fresh perspective. I can see three possible options:
1. Keep the .ini file format and build on top of that. _Pros:_ * no need to write a script to convert the old configuration files to a new format * easy to write for the user _Cons:_ * may become increasingly cumbersome to maintain if the available configuration keys are extended * no support for subgroups within one configuration group, so the configuration file may become cluttered or we'd need to handle this manually in the codebase 2. Switch to toml _Pros:_ * the language support semantically boolean/str/numbers (and parse them accordingly) * language is standardized & similar to .INI * easy to write (but see below) * possible to have nested configs, e.g. to set defaults for a range of different fields:
[yt.fields.gas] # Use linear scale by default for gas fields … use_log = false
[yt.fields.gas.pressure] # … except for pressure fields use_log = true
That would be parsed natively as {'yt': { 'fields': { 'gas': { 'use_log': False, 'pressure':{ 'use_log': True } } } } * the syntax make it explicit what type each value is, so we could check the validity of the config when yt is loaded (for ex. [yt]\nloglevel=nan is invalid because loglevel should be a number). _Cons:_ * users will have to migrate their old config (if any), though we could provide a migration script * yet-another dependency (but we could vendor the parser, there are small implementations) * may be harder to write (how does a user know the expected type of each key? Should [yt]\n loglevel by a string or a number?) * for developers: we'd need to change the way the config is supported in the codebase (currently we're relying on ConfigParser) * there are some gotchas, for example boolean true is written "true" not "True" 3. Switch to yaml or json?
What is your opinion? As a starting point, you could think about how you'd implement the example given above to deactivate the log scale for plots made with any field with type "gas", except pressure (is this something we want? If so, what's the best solution to achieve it in a meaningful way for the user?).
Regards, Corentin
-- Dr. Corentin Cadiou Post Doctoral Research Assistant Cosmoparticle Initiative Hub, desk 27 University College London (UCL) Gower St, Bloomsbury, London WC1E 6BT
mobile: +33.6.43.18.66.83
_______________________________________________ yt-dev mailing list -- yt-dev@python.org <mailto:yt-dev@python.org> To unsubscribe send an email to yt-dev-leave@python.org <mailto:yt-dev-leave@python.org> https://mail.python.org/mailman3/lists/yt-dev.python.org/ <https://mail.python.org/mailman3/lists/yt-dev.python.org/> Member address: matthewturk@gmail.com <mailto:matthewturk@gmail.com>
_______________________________________________ yt-dev mailing list -- yt-dev@python.org <mailto:yt-dev@python.org> To unsubscribe send an email to yt-dev-leave@python.org <mailto:yt-dev-leave@python.org> https://mail.python.org/mailman3/lists/yt-dev.python.org/ <https://mail.python.org/mailman3/lists/yt-dev.python.org/> Member address: brittonsmith@gmail.com <mailto:brittonsmith@gmail.com>
_______________________________________________ yt-dev mailing list -- yt-dev@python.org To unsubscribe send an email to yt-dev-leave@python.org https://mail.python.org/mailman3/lists/yt-dev.python.org/ Member address: corentin.cadiou@iap.fr -- Dr. Corentin Cadiou Post Doctoral Research Assistant Cosmoparticle Initiative Hub, desk 27 University College London (UCL) Gower St, Bloomsbury, London WC1E 6BT
mobile: +33.6.43.18.66.83 <OpenPGP_signature.sig>
_______________________________________________ yt-dev mailing list -- yt-dev@python.org To unsubscribe send an email to yt-dev-leave@python.org https://mail.python.org/mailman3/lists/yt-dev.python.org/ Member address: corentin.cadiou@iap.fr
-- Dr. Corentin Cadiou Post Doctoral Research Assistant Cosmoparticle Initiative Hub, desk 27 University College London (UCL) Gower St, Bloomsbury, London WC1E 6BT mobile: +33.6.43.18.66.83
On 26 Nov 2020, at 17:44, Corentin CADIOU <corentin.cadiou@iap.fr> wrote:
We could of course fix the ambiguity by setting priority rules (ex. in this case, we could decide a field name is more specific than a field type, so it should have priority). What do others think about this?
That’s fine by me for reference, I was merely curious, I don’t have a strong opinion on how it should work.
TLDR; to ease the process of others weighing in, we have three options (I'd actually be in favor of Clement's option):
- do not support local config files, - Clement's proposition: load the local config file (if any), otherwise load the global config file, - Corentin's proposition: load both the local config file (if any) and the global config file + provide a way to dump the current configviaa CLI argument.
More details:
The way the PR is implementedcurrentlyis that it loads the config from ~/.config/yt/yt.toml, then it walks up the folder tree until it finds a yt.toml file. At the first match (if any), it reads it, overrides whatever was defined earlier and stops walking the tree up (i.e. at most one local yt.toml file will ever be read, in the same spirit as git stopping at the first parent folder that contains a .git directory).
If I am correct, your proposition is the following: if any local config file is found, then do not read the global config (~/.config/yt/yt.toml). If we were to adopt this proposition, which file should `yt config set foo bar` configure?
Here’s a suggestion: - if no local config file is present already, store the new param in the global file - else, error out and require that the command be rerun with either `—local` or `—global` I think this covers every possible use case rather simply and don’t put us in a position where the feature is either hard to implement, document, or understand from the user pov.
participants (4)
-
Britton Smith -
Clément Robert -
Corentin CADIOU -
Matthew Turk