Dear yt devs,
TLDR: should we change the configuration format from .ini to another one (e.g. toml)?
Last meeting we discussed what steps were required for the 4.0 release and one of the point was whether we should change how we handle configurations files (e.g. to get https://github.com/yt-project/yt/pull/1931 eventually in). Given that changing the format of yt's configuration is a big breaking change, this would have its place in a yt-3 to 4 transition plan.
Amon the possibilities of a new or improved configuration system, users may be able to set some preferences per-field (e.g. have a configuration that implements the user's wish that "velocity_divergence should be displayed using a linear scale, with the seismic colormap"), or to pick their favourite unit system (e.g. cgs for the astro community, mks for the others).
I know this has already been discussed earlier (both in PR comments and /via/ email), but let's take a moment to think about it with a fresh perspective. I can see three possible options:
- Keep the .ini file format and build on top of that.
_Pros:_
- no need to write a script to convert the old configuration files to a new format
- easy to write for the user _Cons:_
- may become increasingly cumbersome to maintain if the available configuration keys are extended
- no support for subgroups within one configuration group, so the configuration file may become cluttered or we'd need to handle this manually in the codebase
Switch to toml _Pros:_
- the language support semantically boolean/str/numbers (and parse them accordingly)
- language is standardized & similar to .INI
- easy to write (but see below)
possible to have nested configs, e.g. to set defaults for a range of different fields:
[yt.fields.gas] # Use linear scale by default for gas fields … use_log = false
[yt.fields.gas.pressure] # … except for pressure fields use_log = true
That would be parsed natively as {'yt': { 'fields': { 'gas': { 'use_log': False, 'pressure':{ 'use_log': True } } } }
- the syntax make it explicit what type each value is, so we could check the validity of the config when yt is loaded (for ex. [yt]\nloglevel=nan is invalid because loglevel should be a number). _Cons:_
- users will have to migrate their old config (if any), though we could provide a migration script
- yet-another dependency (but we could vendor the parser, there are small implementations)
- may be harder to write (how does a user know the expected type of each key? Should [yt]\n loglevel by a string or a number?)
- for developers: we'd need to change the way the config is supported in the codebase (currently we're relying on ConfigParser)
- there are some gotchas, for example boolean true is written "true" not "True"
- Switch to yaml or json?
What is your opinion? As a starting point, you could think about how you'd implement the example given above to deactivate the log scale for plots made with any field with type "gas", except pressure (is this something we want? If so, what's the best solution to achieve it in a meaningful way for the user?).
Regards, Corentin
-- Dr. Corentin Cadiou Post Doctoral Research Assistant Cosmoparticle Initiative Hub, desk 27 University College London (UCL) Gower St, Bloomsbury, London WC1E 6BT
mobile: +33.6.43.18.66.83
Hi Corentin,
This is a great idea.
I was originally opposed to toml. But, I think it's now a transitive dependency for yt anyway, so we should be safe using it, and I think that it's reasonably common enough that it will be usable for folks. I would wager than migration is not going to be too terribly difficult, as we can probably parse a config, see if it matches the defaults, and if not, just bail and say that it should be converted.
On Mon, Nov 23, 2020 at 8:40 AM Corentin CADIOU corentin.cadiou@iap.fr wrote:
Dear yt devs,
TLDR: should we change the configuration format from .ini to another one
(e.g. toml)?
Last meeting we discussed what steps were required for the 4.0 release and one of the point was whether we should change how we handle configurations files (e.g. to get https://github.com/yt-project/yt/pull/1931 eventually in). Given that changing the format of yt's configuration is a big breaking change, this would have its place in a yt-3 to 4 transition plan.
Amon the possibilities of a new or improved configuration system, users may be able to set some preferences per-field (e.g. have a configuration that implements the user's wish that "velocity_divergence should be displayed using a linear scale, with the seismic colormap"), or to pick their favourite unit system (e.g. cgs for the astro community, mks for the others).
I know this has already been discussed earlier (both in PR comments and via email), but let's take a moment to think about it with a fresh perspective. I can see three possible options:
- Keep the .ini file format and build on top of that. Pros:
- no need to write a script to convert the old configuration files to
a new format
- easy to write for the user Cons:
- may become increasingly cumbersome to maintain if the available configuration keys are extended
- no support for subgroups within one configuration group, so the configuration file may become cluttered or we'd need to handle this manually in the codebase
- Switch to toml Pros:
the language support semantically boolean/str/numbers (and parse them accordingly)
- language is standardized & similar to .INI
- easy to write (but see below)
possible to have nested configs, e.g. to set defaults for a range of different fields:
[yt.fields.gas] # Use linear scale by default for gas fields … use_log = false
[yt.fields.gas.pressure] # … except for pressure fields use_log = true
That would be parsed natively as {'yt': { 'fields': {
'gas': { 'use_log': False, 'pressure': { 'use_log': True } }} }
- the syntax make it explicit what type each value is, so we could check the validity of the config when yt is loaded (for ex. [yt]\nloglevel=nan is invalid because loglevel should be a number). Cons:
- users will have to migrate their old config (if any), though we
could provide a migration script
- yet-another dependency (but we could vendor the parser, there are small implementations)
- may be harder to write (how does a user know the expected type of each key? Should [yt]\n loglevel by a string or a number?)
- for developers: we'd need to change the way the config is supported in the codebase (currently we're relying on ConfigParser)
- there are some gotchas, for example boolean true is written "true" not "True"
- Switch to yaml or json?
What is your opinion? As a starting point, you could think about how you'd implement the example given above to deactivate the log scale for plots made with any field with type "gas", except pressure (is this something we want? If so, what's the best solution to achieve it in a meaningful way for the user?).
Regards, Corentin
-- Dr. Corentin Cadiou Post Doctoral Research Assistant Cosmoparticle Initiative Hub, desk 27 University College London (UCL) Gower St, Bloomsbury, London WC1E 6BT
mobile: +33.6.43.18.66.83
yt-dev mailing list -- yt-dev@python.org To unsubscribe send an email to yt-dev-leave@python.org https://mail.python.org/mailman3/lists/yt-dev.python.org/ Member address: matthewturk@gmail.com
Hi, I support changing the format to toml, which feels infinitely less controversial than yaml or json since it’s already so close to ini. Correct me if I’m wrong, but that would allow for the config to be accessed internally as a dictionary instead of through a ConfigParser instance, right ?
I also support Matt’s suggestion for migration strategy, provided we don’t expect silent parsing errors to occur (e.g. True being parsed as a string instead of a boolean. This example is not relevant but you get the idea).
Cheers, Clément
On 23 Nov 2020, at 16:05, Matthew Turk matthewturk@gmail.com wrote:
Hi Corentin,
This is a great idea.
I was originally opposed to toml. But, I think it's now a transitive dependency for yt anyway, so we should be safe using it, and I think that it's reasonably common enough that it will be usable for folks. I would wager than migration is not going to be too terribly difficult, as we can probably parse a config, see if it matches the defaults, and if not, just bail and say that it should be converted.
On Mon, Nov 23, 2020 at 8:40 AM Corentin CADIOU corentin.cadiou@iap.fr wrote:
Dear yt devs,
TLDR: should we change the configuration format from .ini to another one (e.g. toml)?
Last meeting we discussed what steps were required for the 4.0 release and one of the point was whether we should change how we handle configurations files (e.g. to get https://github.com/yt-project/yt/pull/1931 eventually in). Given that changing the format of yt's configuration is a big breaking change, this would have its place in a yt-3 to 4 transition plan.
Amon the possibilities of a new or improved configuration system, users may be able to set some preferences per-field (e.g. have a configuration that implements the user's wish that "velocity_divergence should be displayed using a linear scale, with the seismic colormap"), or to pick their favourite unit system (e.g. cgs for the astro community, mks for the others).
I know this has already been discussed earlier (both in PR comments and via email), but let's take a moment to think about it with a fresh perspective. I can see three possible options:
Keep the .ini file format and build on top of that. Pros:
no need to write a script to convert the old configuration files to a new format
easy to write for the user Cons:
may become increasingly cumbersome to maintain if the available configuration keys are extended
- no support for subgroups within one configuration group, so the configuration file may become cluttered or we'd need to handle this manually in the codebase
Switch to toml Pros:
the language support semantically boolean/str/numbers (and parse them accordingly)
- language is standardized & similar to .INI
- easy to write (but see below)
- possible to have nested configs, e.g. to set defaults for a range of different fields:
[yt.fields.gas]
# Use linear scale by default for gas fields …
use_log = false
[yt.fields.gas.pressure]
# … except for pressure fields
use_log = true
That would be parsed natively as {'yt': { 'fields': {'gas': {'use_log': False,'pressure':{ 'use_log': True }}} }
the syntax make it explicit what type each value is, so we could check the validity of the config when yt is loaded (for ex. [yt]\nloglevel=nan is invalid because loglevel should be a number). Cons:
users will have to migrate their old config (if any), though we could provide a migration script
- yet-another dependency (but we could vendor the parser, there are small implementations)
- may be harder to write (how does a user know the expected type of each key? Should [yt]\n loglevel by a string or a number?)
- for developers: we'd need to change the way the config is supported in the codebase (currently we're relying on ConfigParser)
there are some gotchas, for example boolean true is written "true" not "True"
Switch to yaml or json?
What is your opinion? As a starting point, you could think about how you'd implement the example given above to deactivate the log scale for plots made with any field with type "gas", except pressure (is this something we want? If so, what's the best solution to achieve it in a meaningful way for the user?).
Regards, Corentin
-- Dr. Corentin Cadiou Post Doctoral Research Assistant Cosmoparticle Initiative Hub, desk 27 University College London (UCL) Gower St, Bloomsbury, London WC1E 6BT
mobile: +33.6.43.18.66.83
yt-dev mailing list -- yt-dev@python.org To unsubscribe send an email to yt-dev-leave@python.org https://mail.python.org/mailman3/lists/yt-dev.python.org/ Member address: matthewturk@gmail.com
Hi there,
I support the idea of providing a conversion script and printing out a command to run to do the conversion. This needs to be a disruption that takes people less than 60 second to return to their workflow.
Britton
On Mon, Nov 23, 2020 at 8:53 PM Clément Robert via yt-dev yt-dev@python.org wrote:
Hi, I support changing the format to toml, which feels infinitely less controversial than yaml or json since it’s already so close to ini. Correct me if I’m wrong, but that would allow for the config to be accessed internally as a dictionary instead of through a ConfigParser instance, right ?
I also support Matt’s suggestion for migration strategy, provided we don’t expect silent parsing errors to occur (e.g. True being parsed as a string instead of a boolean. This example is not relevant but you get the idea).
Cheers, Clément
On 23 Nov 2020, at 16:05, Matthew Turk matthewturk@gmail.com wrote:
Hi Corentin,
This is a great idea.
I was originally opposed to toml. But, I think it's now a transitive dependency for yt anyway, so we should be safe using it, and I think that it's reasonably common enough that it will be usable for folks. I would wager than migration is not going to be too terribly difficult, as we can probably parse a config, see if it matches the defaults, and if not, just bail and say that it should be converted.
On Mon, Nov 23, 2020 at 8:40 AM Corentin CADIOU corentin.cadiou@iap.fr wrote:
Dear yt devs,
TLDR: should we change the configuration format from .ini to another one
(e.g. toml)?
Last meeting we discussed what steps were required for the 4.0 release and one of the point was whether we should change how we handle configurations files (e.g. to get https://github.com/yt-project/yt/pull/1931 eventually in). Given that changing the format of yt's configuration is a big breaking change, this would have its place in a yt-3 to 4 transition plan.
Amon the possibilities of a new or improved configuration system, users may be able to set some preferences per-field (e.g. have a configuration that implements the user's wish that "velocity_divergence should be displayed using a linear scale, with the seismic colormap"), or to pick their favourite unit system (e.g. cgs for the astro community, mks for the others).
I know this has already been discussed earlier (both in PR comments and via email), but let's take a moment to think about it with a fresh perspective. I can see three possible options:
- Keep the .ini file format and build on top of that. Pros:
- no need to write a script to convert the old configuration files to
a new format
- easy to write for the user Cons:
- may become increasingly cumbersome to maintain if the available configuration keys are extended
- no support for subgroups within one configuration group, so the configuration file may become cluttered or we'd need to handle this manually in the codebase
- Switch to toml Pros:
the language support semantically boolean/str/numbers (and parse them accordingly)
- language is standardized & similar to .INI
- easy to write (but see below)
possible to have nested configs, e.g. to set defaults for a range of different fields:
[yt.fields.gas] # Use linear scale by default for gas fields … use_log = false
[yt.fields.gas.pressure] # … except for pressure fields use_log = true
That would be parsed natively as {'yt': { 'fields': {
'gas': { 'use_log': False, 'pressure': { 'use_log': True } }} }
- the syntax make it explicit what type each value is, so we could check the validity of the config when yt is loaded (for ex. [yt]\nloglevel=nan is invalid because loglevel should be a number). Cons:
- users will have to migrate their old config (if any), though we
could provide a migration script
- yet-another dependency (but we could vendor the parser, there are small implementations)
- may be harder to write (how does a user know the expected type of each key? Should [yt]\n loglevel by a string or a number?)
- for developers: we'd need to change the way the config is supported in the codebase (currently we're relying on ConfigParser)
- there are some gotchas, for example boolean true is written "true" not "True"
- Switch to yaml or json?
What is your opinion? As a starting point, you could think about how you'd implement the example given above to deactivate the log scale for plots made with any field with type "gas", except pressure (is this something we want? If so, what's the best solution to achieve it in a meaningful way for the user?).
Regards, Corentin
-- Dr. Corentin Cadiou Post Doctoral Research Assistant Cosmoparticle Initiative Hub, desk 27 University College London (UCL) Gower St, Bloomsbury, London WC1E 6BT
mobile: +33.6.43.18.66.83
yt-dev mailing list -- yt-dev@python.org To unsubscribe send an email to yt-dev-leave@python.org https://mail.python.org/mailman3/lists/yt-dev.python.org/ Member address: matthewturk@gmail.com
yt-dev mailing list -- yt-dev@python.org To unsubscribe send an email to yt-dev-leave@python.org https://mail.python.org/mailman3/lists/yt-dev.python.org/ Member address: brittonsmith@gmail.com
Dear all,
Thanks for the constructive feedback. I have hopefully taken all your comments into account and moved our configuration to toml. The pull request is here https://github.com/yt-project/yt/pull/2981.
Following Britton's comment, I tried to keep the migration as simple as
possible. It can be achieved by typing yt config migrate once. Note
that toml is case-sensitive, while ConfigParser is not. To accommodate
for this, the migration script will convert the case of known variables
(ex. [yt] LOGLEVEL= 10 will be converted to [yt] loglevel = 10).
Now, here are the novelties:
- the format is now toml.
- the configuration is loaded from ~/.config/yt/yt.toml (instead of
~/config/yt/ytrc), and if a local yt.toml file is found in the
current folder or any of its parent, it is also loaded (and
overrides previous preferences). You can use
yt config print-path --localto ask yt where it found the local configuration file. - the local configuration file can be configured from the CLI in the
same way as the global one, provided you add the
--localflag, e.g.yt config set --local yt loglevel 1will set the loglevel to 1 in the local config and leave the global config unchanged. If no local config is found, a new one is created in the working directory. the configuration is now type-sensitive (a useful message is thrown at the user if the type changes). See for example
import yt yt.config.ytcfg["foo", "bar", "baz"] = 10 yt.config.ytcfg["foo", "bar", "baz"] = "I cannot pass a string now" [...] TypeError: Error when setting foo.bar.baz. Tried to assign a value of type <class 'str'>, expected type <class 'int'>. This entry was last modified in runtime. yt.config.ytcfg["yt", "loglevel"] = "invalid loglevel" [...] TypeError: Error when setting yt.loglevel. Tried to assign a value of type <class 'str'>, expected type <class 'int'>. This entry was last modified in file: /path/to/yt/installation/yt/yt.toml.
This allows errors to be caught as soon as possible instead of the issue popping randomly when the actual configuration entry is used for the first time.
I added the possibility to find the most specific entry matching a given path. Once again, let me illustrate that with an example: imagine you have this configuration
[fields] cmap = "arbre"
[fields.gas] cmap = "viridis"
[fields.gas.temperature] cmap = "plasma"
then you can use
ytcfg.get_most_specificto find the most specific option corresponding to your request. For exampleyt.config.ytcfg.get_most_specific("fields", "gas", "temperature", "cmap") # full match with fields.gas.temperature:cmap 'plasma' yt.config.ytcfg.get_most_specific("fields", "gas", "density", "cmap") # partial match with fields.gas:cmap 'viridis' yt.config.ytcfg.get_most_specific("fields", "io", "particle_velocity", "cmap") # partial match with fields:cmap 'arbre'
This enable us to set options for example for all fields, for all fields of type
gasand for specific fields individually.
The PR is ready to be merged, but I am very much open to suggestions of things to add or remove before it gets in.
Cheers, Corentin
On 24/11/2020 10:01, Britton Smith wrote:
Hi there,
I support the idea of providing a conversion script and printing out a command to run to do the conversion. This needs to be a disruption that takes people less than 60 second to return to their workflow.
Britton
On Mon, Nov 23, 2020 at 8:53 PM Clément Robert via yt-dev
<yt-dev@python.org mailto:yt-dev@python.org> wrote:
Hi,
I support changing the format to toml, which feels infinitely less
controversial than yaml or json since it’s already so close to ini.
Correct me if I’m wrong, but that would allow for the config to be
accessed internally as a dictionary instead of through a
ConfigParser instance, right ?
I also support Matt’s suggestion for migration strategy, provided
we don’t expect silent parsing errors to occur (e.g. True being
parsed as a string instead of a boolean. This example is not
relevant but you get the idea).
Cheers,
Clément
On 23 Nov 2020, at 16:05, Matthew Turk
<matthewturk@gmail.com
<mailto:matthewturk@gmail.com>> wrote:
Hi Corentin,
This is a great idea.
I was originally opposed to toml. But, I think it's now a
transitive dependency for yt anyway, so we should be safe using
it, and I think that it's reasonably common enough that it will
be usable for folks. I would wager than migration is not going
to be too terribly difficult, as we can probably parse a config,
see if it matches the defaults, and if not, just bail and say
that it should be converted.
On Mon, Nov 23, 2020 at 8:40 AM Corentin CADIOU
<corentin.cadiou@iap.fr <mailto:corentin.cadiou@iap.fr>> wrote:
Dear yt devs,
TLDR: should we change the configuration format from .ini to
another one (e.g. toml)?
------------------------------------------------------------------------
Last meeting we discussed what steps were required for the
4.0 release and one of the point was whether we should change
how we handle configurations files (e.g. to get
https://github.com/yt-project/yt/pull/1931
<https://github.com/yt-project/yt/pull/1931> eventually in).
Given that changing the format of yt's configuration is a big
breaking change, this would have its place in a yt-3 to 4
transition plan.
Amon the possibilities of a new or improved configuration
system, users may be able to set some preferences per-field
(e.g. have a configuration that implements the user's wish
that "velocity_divergence should be displayed using a linear
scale, with the seismic colormap"), or to pick their
favourite unit system (e.g. cgs for the astro community, mks
for the others).
I know this has already been discussed earlier (both in PR
comments and /via/ email), but let's take a moment to think
about it with a fresh perspective. I can see three possible
options:
1. Keep the .ini file format and build on top of that.
_Pros:_
* no need to write a script to convert the old
configuration files to a new format
* easy to write for the user
_Cons:_
* may become increasingly cumbersome to maintain if the
available configuration keys are extended
* no support for subgroups within one configuration
group, so the configuration file may become cluttered
or we'd need to handle this manually in the codebase
2. Switch to toml
_Pros:_
* the language support semantically boolean/str/numbers
(and parse them accordingly)
* language is standardized & similar to .INI
* easy to write (but see below)
* possible to have nested configs, e.g. to set defaults
for a range of different fields:
[yt.fields.gas]
# Use linear scale by default for gas fields …
use_log = false
[yt.fields.gas.pressure]
# … except for pressure fields
use_log = true
That would be parsed natively as
{'yt': {
'fields': {
'gas': {
'use_log': False,
'pressure':{
'use_log': True
}
}
}
}
* the syntax make it explicit what type each value is,
so we could check the validity of the config when yt
is loaded (for ex. [yt]\nloglevel=nan is invalid
because loglevel should be a number).
_Cons:_
* users will have to migrate their old config (if any),
though we could provide a migration script
* yet-another dependency (but we could vendor the
parser, there are small implementations)
* may be harder to write (how does a user know the
expected type of each key? Should [yt]\n loglevel by
a string or a number?)
* for developers: we'd need to change the way the
config is supported in the codebase (currently we're
relying on ConfigParser)
* there are some gotchas, for example boolean true is
written "true" not "True"
3. Switch to yaml or json?
What is your opinion? As a starting point, you could think
about how you'd implement the example given above to
deactivate the log scale for plots made with any field with
type "gas", except pressure (is this something we want? If
so, what's the best solution to achieve it in a meaningful
way for the user?).
Regards,
Corentin
--
Dr. Corentin Cadiou
Post Doctoral Research Assistant
Cosmoparticle Initiative Hub, desk 27
University College London (UCL)
Gower St, Bloomsbury, London WC1E 6BT
mobile: +33.6.43.18.66.83
_______________________________________________
yt-dev mailing list -- yt-dev@python.org
<mailto:yt-dev@python.org>
To unsubscribe send an email to yt-dev-leave@python.org
<mailto:yt-dev-leave@python.org>
https://mail.python.org/mailman3/lists/yt-dev.python.org/
<https://mail.python.org/mailman3/lists/yt-dev.python.org/>
Member address: matthewturk@gmail.com
<mailto:matthewturk@gmail.com>
_______________________________________________
yt-dev mailing list -- yt-dev@python.org <mailto:yt-dev@python.org>
To unsubscribe send an email to yt-dev-leave@python.org
<mailto:yt-dev-leave@python.org>
https://mail.python.org/mailman3/lists/yt-dev.python.org/
<https://mail.python.org/mailman3/lists/yt-dev.python.org/>
Member address: brittonsmith@gmail.com <mailto:brittonsmith@gmail.com>
yt-dev mailing list -- yt-dev@python.org To unsubscribe send an email to yt-dev-leave@python.org https://mail.python.org/mailman3/lists/yt-dev.python.org/ Member address: corentin.cadiou@iap.fr
-- Dr. Corentin Cadiou Post Doctoral Research Assistant Cosmoparticle Initiative Hub, desk 27 University College London (UCL) Gower St, Bloomsbury, London WC1E 6BT
mobile: +33.6.43.18.66.83
Hi Corentin, thanks a lot for getting so much done in that direction so quickly.
Regarding the get_most_specific feature, I’m wondering how flexible it is.
For instance, can I specify a cmap for all “temperature” fields, independently from the
ftype ? I imagine it would read something like
[fields.temperature] cmap = “plasma”
Another point I’d like to raise, and it’s not mentioned in your email (but we discussed it in private), is the behaviour in case more than one config file is found (in cwd and the global one for instance). So currently, according to our discussion, both config files would be taken into account, the local one taking priority in case of overlapping. I can imagine that one would like to build nested configurations from nested directories in a project (although I _hope_ no-one actually thinks this a good idea !), but what I’m worried about is the reproducibility among participants in a shared project. I wrote this issue a few month back https://github.com/yt-project/yt/issues/2635 Basically, allowing local config files enable them to be shared along with a notebook or script in a repo, but if the global config file is still having an effect, then the results may vary from one machine to the other and I don’t think that’s a desirable trait in general. On the other hand, it would be nice that the possibility remains available, but I think it should only be used if and when this behaviour is explicitly asked for. In other words, this could be a parameter defined in the local-most file !
Clément
On 26 Nov 2020, at 16:03, Corentin CADIOU corentin.cadiou@iap.fr wrote:
Dear all,
Thanks for the constructive feedback. I have hopefully taken all your comments into account and moved our configuration to toml. The pull request is here https://github.com/yt-project/yt/pull/2981.
Following Britton's comment, I tried to keep the migration as simple as possible. It
can be achieved by typing yt config migrate once. Note that toml is
case-sensitive, while ConfigParser is not. To accommodate for this, the migration script
will convert the case of known variables (ex. [yt] LOGLEVEL= 10 will be converted to [yt]
loglevel = 10).
Now, here are the novelties:
- the format is now toml.
- the configuration is loaded from ~/.config/yt/yt.toml (instead of ~/config/yt/ytrc),
and if a local yt.toml file is found in the current folder or any of its parent, it is
also loaded (and overrides previous preferences). You can use
yt config print-path --localto ask yt where it found the local configuration file. - the local configuration file can be configured from the CLI in the same way as the
global one, provided you add the
--localflag, e.g.yt config set --local yt loglevel 1will set the loglevel to 1 in the local config and leave the global config unchanged. If no local config is found, a new one is created in the working directory. - the configuration is now type-sensitive (a useful message is thrown at the user if the type changes). See for example
import yt yt.config.ytcfg["foo", "bar", "baz"] = 10 yt.config.ytcfg["foo", "bar", "baz"] = "I cannot pass a string now" [...] TypeError: Error when setting foo.bar.baz. Tried to assign a value of type <class 'str'>, expected type <class 'int'>. This entry was last modified in runtime. yt.config.ytcfg["yt", "loglevel"] = "invalid loglevel" [...] TypeError: Error when setting yt.loglevel. Tried to assign a value of type <class 'str'>, expected type <class 'int'>. This entry was last modified in file: /path/to/yt/installation/yt/yt.toml. This allows errors to be caught as soon as possible instead of the issue popping randomly when the actual configuration entry is used for the first time.
- I added the possibility to find the most specific entry matching a given path. Once again, let me illustrate that with an example: imagine you have this configuration
[fields] cmap = "arbre"
[fields.gas] cmap = "viridis"
[fields.gas.temperature] cmap = "plasma"
then you can use ytcfg.get_most_specific to find the most specific option
corresponding to your request. For example
yt.config.ytcfg.get_most_specific("fields",
"gas", "temperature", "cmap") # full match with fields.gas.temperature:cmap
'plasma'
yt.config.ytcfg.get_most_specific("fields", "gas", "density", "cmap") # partial match with
fields.gas:cmap
'viridis'
yt.config.ytcfg.get_most_specific("fields", "io", "particle_velocity", "cmap") # partial
match with fields:cmap
'arbre'
This enable us to set options for example for all fields, for all fields of type
gas and for specific fields individually.
The PR is ready to be merged, but I am very much open to suggestions of things to add or remove before it gets in.
Cheers, Corentin
On 24/11/2020 10:01, Britton Smith wrote:
Hi there,
I support the idea of providing a conversion script and printing out a command to run to do the conversion. This needs to be a disruption that takes people less than 60 second to return to their workflow.
Britton
On Mon, Nov 23, 2020 at 8:53 PM Clément Robert via yt-dev yt-dev@python.org wrote:
Hi, I support changing the format to toml, which feels infinitely less controversial than yaml or json since it’s already so close to ini. Correct me if I’m wrong, but that would allow for the config to be accessed internally as a dictionary instead of through a ConfigParser instance, right ?
I also support Matt’s suggestion for migration strategy, provided we don’t expect silent parsing errors to occur (e.g. True being parsed as a string instead of a boolean. This example is not relevant but you get the idea).
Cheers, Clément
On 23 Nov 2020, at 16:05, Matthew Turk matthewturk@gmail.com wrote:
Hi Corentin,
This is a great idea.
I was originally opposed to toml. But, I think it's now a transitive dependency for yt anyway, so we should be safe using it, and I think that it's reasonably common enough that it will be usable for folks. I would wager than migration is not going to be too terribly difficult, as we can probably parse a config, see if it matches the defaults, and if not, just bail and say that it should be converted.
On Mon, Nov 23, 2020 at 8:40 AM Corentin CADIOU corentin.cadiou@iap.fr wrote:
Dear yt devs,
TLDR: should we change the configuration format from .ini to another one (e.g. toml)?
Last meeting we discussed what steps were required for the 4.0 release and one of the point was whether we should change how we handle configurations files (e.g. to get https://github.com/yt-project/yt/pull/1931 eventually in). Given that changing the format of yt's configuration is a big breaking change, this would have its place in a yt-3 to 4 transition plan.
Amon the possibilities of a new or improved configuration system, users may be able to set some preferences per-field (e.g. have a configuration that implements the user's wish that "velocity_divergence should be displayed using a linear scale, with the seismic colormap"), or to pick their favourite unit system (e.g. cgs for the astro community, mks for the others).
I know this has already been discussed earlier (both in PR comments and via email), but let's take a moment to think about it with a fresh perspective. I can see three possible options:
Keep the .ini file format and build on top of that. Pros:
no need to write a script to convert the old configuration files to a new format
easy to write for the user Cons:
may become increasingly cumbersome to maintain if the available configuration keys are extended
- no support for subgroups within one configuration group, so the configuration file may become cluttered or we'd need to handle this manually in the codebase
Switch to toml Pros:
the language support semantically boolean/str/numbers (and parse them accordingly)
- language is standardized & similar to .INI
- easy to write (but see below)
- possible to have nested configs, e.g. to set defaults for a range of different fields:
[yt.fields.gas]
# Use linear scale by default for gas fields …
use_log = false
[yt.fields.gas.pressure]
# … except for pressure fields
use_log = true
That would be parsed natively as {'yt': { 'fields': {'gas': {'use_log': False,'pressure':{ 'use_log': True }}} }
the syntax make it explicit what type each value is, so we could check the validity of the config when yt is loaded (for ex. [yt]\nloglevel=nan is invalid because loglevel should be a number). Cons:
users will have to migrate their old config (if any), though we could provide a migration script
- yet-another dependency (but we could vendor the parser, there are small implementations)
- may be harder to write (how does a user know the expected type of each key? Should [yt]\n loglevel by a string or a number?)
- for developers: we'd need to change the way the config is supported in the codebase (currently we're relying on ConfigParser)
there are some gotchas, for example boolean true is written "true" not "True"
Switch to yaml or json?
What is your opinion? As a starting point, you could think about how you'd implement the example given above to deactivate the log scale for plots made with any field with type "gas", except pressure (is this something we want? If so, what's the best solution to achieve it in a meaningful way for the user?).
Regards, Corentin
-- Dr. Corentin Cadiou Post Doctoral Research Assistant Cosmoparticle Initiative Hub, desk 27 University College London (UCL) Gower St, Bloomsbury, London WC1E 6BT
mobile: +33.6.43.18.66.83
yt-dev mailing list -- yt-dev@python.org To unsubscribe send an email to yt-dev-leave@python.org https://mail.python.org/mailman3/lists/yt-dev.python.org/ Member address: matthewturk@gmail.com
yt-dev mailing list -- yt-dev@python.org To unsubscribe send an email to yt-dev-leave@python.org https://mail.python.org/mailman3/lists/yt-dev.python.org/ Member address: brittonsmith@gmail.com
yt-dev mailing list -- yt-dev@python.org To unsubscribe send an email to yt-dev-leave@python.org
https://mail.python.org/mailman3/lists/yt-dev.python.org/ Member address: corentin.cadiou@iap.fr
-- Dr. Corentin Cadiou Post Doctoral Research Assistant Cosmoparticle Initiative Hub, desk 27 University College London (UCL) Gower St, Bloomsbury, London WC1E 6BT
mobile: +33.6.43.18.66.83
<OpenPGP_signature.sig>
Hi Clement,
On 26/11/2020 16:01, Clément Robert via yt-dev wrote:
Hi Corentin, thanks a lot for getting so much done in
that direction
so quickly.
Regarding the get_most_specific feature, I’m wondering how flexible
it is. For instance, can I specify a cmap for all “temperature”
fields, independently from the ftype ? I imagine it would read
something like
[fields.temperature] cmap = “plasma”
Currently, the get_most_specific feature finds the longest
consecutive
common path between the config and the requested config item, starting
from the left only. In other words, if you have
[fields.temperature]
cmap = "plasma"
it will return "plasma" if you call ytcfg.get_most_specific("fields",
"temperature", "cmap") but will fail with
ytcfg.get_most_specific("fields", "gas", "temperature", "cmap"). The
reason is that otherwise we may end up with ambiguous results, i.e. what
should ytcfg.get_most_specific("fields", "gas", "temperature", "cmap")
return with the following config?
[fields.temperature]
cmap = "plasma"
[fields.gas]
cmap = "not_plasma"
We could of course fix the ambiguity by setting priority rules (ex. in this case, we could decide a field name is more specific than a field type, so it should have priority). What do others think about this?
>
Another point I’d like to raise, and it’s not mentioned in your email (but we discussed it in private), is the behaviour in case more than one config file is found (in cwd and the global one for instance). So currently, according to our discussion, both config files would be taken into account, the local one taking priority in case of overlapping. I can imagine that one would like to build nested configurations from nested directories in a project (although I _hope_ no-one actually thinks this a good idea !), but what I’m worried about is the reproducibility among participants in a shared project. I wrote this issue a few month back https://github.com/yt-project/yt/issues/2635 https://github.com/yt-project/yt/issues/2635 Basically, allowing local config files enable them to be shared along with a notebook or script in a repo, but if the global config file is still having an effect, then the results may vary from one machine to the other and I don’t think that’s a desirable trait in general. On the other hand, it would be nice that the possibility remains available, but I think it should only be used if and when this behaviour is explicitly asked for. In other words, this could be a parameter defined in the local-most file !
TLDR; to ease the process of others weighing in, we have three options (I'd actually be in favor of Clement's option):
- do not support local config files,
- Clement's proposition: load the local config file (if any), otherwise load the global config file,
- Corentin's proposition: load both the local config file (if any) and the global config file + provide a way to dump the current config /via/ a CLI argument.
More details:
The way the PR is implemented /currently/ is that it loads the config from ~/.config/yt/yt.toml, then it walks up the folder tree until it finds a yt.toml file. At the first match (if any), it reads it, overrides whatever was defined earlier and stops walking the tree up (i.e. at most one local yt.toml file will ever be read, in the same spirit as git stopping at the first parent folder that contains a .git directory).
If I am correct, your proposition is the following: if any local config
file is found, then do not read the global config
(~/.config/yt/yt.toml). If we were to adopt this proposition, which file
should yt config set foo bar configure?
Another option would be to read both (as is currently the case in my
PR), but have the cli option yt config list dump the full config
(either global + local, or defaults + global + local). Currently, yt
config list shows only the global config and yt config list --local
only the local one.
>
Clément
On 26 Nov 2020, at 16:03, Corentin CADIOU <corentin.cadiou@iap.fr
mailto:corentin.cadiou@iap.fr> wrote:
Dear all,
Thanks for the constructive feedback. I have hopefully taken all your comments into account and moved our configuration to toml. The pull request is here https://github.com/yt-project/yt/pull/2981.
Following Britton's comment, I tried to keep the migration as simple
as possible. It can be achieved by typing yt config migrate once.
Note that toml is case-sensitive, while ConfigParser is not. To
accommodate for this, the migration script will convert the case of
known variables (ex. [yt] LOGLEVEL= 10 will be converted to [yt]
loglevel = 10).
Now, here are the novelties:
- the format is now toml.
- the configuration is loaded from ~/.config/yt/yt.toml (instead of
~/config/yt/ytrc), and if a local yt.toml file is found in the
current folder or any of its parent, it is also loaded (and
overrides previous preferences). You can use
yt config print-path --localto ask yt where it found the local configuration file. - the local configuration file can be configured from the CLI in
the same way as the global one, provided you add the
--localflag, e.g.yt config set --local yt loglevel 1will set the loglevel to 1 in the local config and leave the global config unchanged. If no local config is found, a new one is created in the working directory. the configuration is now type-sensitive (a useful message is thrown at the user if the type changes). See for example
import yt yt.config.ytcfg["foo", "bar", "baz"] = 10 yt.config.ytcfg["foo", "bar", "baz"] = "I cannot pass a string now" [...] TypeError: Error when setting foo.bar.baz. Tried to assign a value of type <class 'str'>, expected type
<class 'int'>. This entry was last modified in runtime. yt.config.ytcfg["yt", "loglevel"] = "invalid loglevel" [...] TypeError: Error when setting yt.loglevel. Tried to assign a value of type <class 'str'>, expected type
<class 'int'>. This entry was last modified in file: /path/to/yt/installation/yt/yt.toml.
This allows errors to be caught as soon as possible instead of the issue popping randomly when the actual configuration entry is used for the first time.
I added the possibility to find the most specific entry matching a given path. Once again, let me illustrate that with an example: imagine you have this configuration
[fields] cmap = "arbre"
[fields.gas] cmap = "viridis"
[fields.gas.temperature] cmap = "plasma"
then you can use
ytcfg.get_most_specificto find the most specific option corresponding to your request. For exampleyt.config.ytcfg.get_most_specific("fields", "gas", "temperature", "cmap") # full match with fields.gas.temperature:cmap 'plasma' yt.config.ytcfg.get_most_specific("fields", "gas", "density", "cmap") # partial match with fields.gas:cmap 'viridis' yt.config.ytcfg.get_most_specific("fields", "io", "particle_velocity", "cmap") # partial match with fields:cmap 'arbre'
This enable us to set options for example for all fields, for all fields of type
gasand for specific fields individually.
The PR is ready to be merged, but I am very much open to suggestions of things to add or remove before it gets in.
Cheers, Corentin
On 24/11/2020 10:01, Britton Smith wrote:
Hi there,
I support the idea of providing a conversion script and printing out a command to run to do the conversion. This needs to be a disruption that takes people less than 60 second to return to their workflow.
Britton
On Mon, Nov 23, 2020 at 8:53 PM Clément Robert via yt-dev
<yt-dev@python.org mailto:yt-dev@python.org> wrote:
Hi,
I support changing the format to toml, which feels infinitely
less controversial than yaml or json since it’s already so close
to ini.
Correct me if I’m wrong, but that would allow for the config to
be accessed internally as a dictionary instead of through a
ConfigParser instance, right ?
I also support Matt’s suggestion for migration strategy,
provided we don’t expect silent parsing errors to occur (e.g.
True being parsed as a string instead of a boolean. This example
is not relevant but you get the idea).
Cheers,
Clément
On 23 Nov 2020, at 16:05, Matthew Turk
<matthewturk@gmail.com
<mailto:matthewturk@gmail.com>> wrote:
Hi Corentin,
This is a great idea.
I was originally opposed to toml. But, I think it's now a
transitive dependency for yt anyway, so we should be safe using
it, and I think that it's reasonably common enough that it will
be usable for folks. I would wager than migration is not going
to be too terribly difficult, as we can probably parse a
config, see if it matches the defaults, and if not, just bail
and say that it should be converted.
On Mon, Nov 23, 2020 at 8:40 AM Corentin CADIOU
<corentin.cadiou@iap.fr <mailto:corentin.cadiou@iap.fr>> wrote:
Dear yt devs,
TLDR: should we change the configuration format from .ini
to another one (e.g. toml)?
------------------------------------------------------------------------
Last meeting we discussed what steps were required for the
4.0 release and one of the point was whether we should
change how we handle configurations files (e.g. to get
https://github.com/yt-project/yt/pull/1931
<https://github.com/yt-project/yt/pull/1931> eventually
in). Given that changing the format of yt's configuration
is a big breaking change, this would have its place in a
yt-3 to 4 transition plan.
Amon the possibilities of a new or improved configuration
system, users may be able to set some preferences per-field
(e.g. have a configuration that implements the user's wish
that "velocity_divergence should be displayed using a
linear scale, with the seismic colormap"), or to pick their
favourite unit system (e.g. cgs for the astro community,
mks for the others).
I know this has already been discussed earlier (both in PR
comments and /via/ email), but let's take a moment to think
about it with a fresh perspective. I can see three possible
options:
1. Keep the .ini file format and build on top of that.
_Pros:_
* no need to write a script to convert the old
configuration files to a new format
* easy to write for the user
_Cons:_
* may become increasingly cumbersome to maintain if
the available configuration keys are extended
* no support for subgroups within one configuration
group, so the configuration file may become
cluttered or we'd need to handle this manually in
the codebase
2. Switch to toml
_Pros:_
* the language support semantically
boolean/str/numbers (and parse them accordingly)
* language is standardized & similar to .INI
* easy to write (but see below)
* possible to have nested configs, e.g. to set
defaults for a range of different fields:
[yt.fields.gas]
# Use linear scale by default for gas fields …
use_log = false
[yt.fields.gas.pressure]
# … except for pressure fields
use_log = true
That would be parsed natively as
{'yt': {
'fields': {
'gas': {
'use_log': False,
'pressure':{
'use_log': True
}
}
}
}
* the syntax make it explicit what type each value
is, so we could check the validity of the config
when yt is loaded (for ex. [yt]\nloglevel=nan is
invalid because loglevel should be a number).
_Cons:_
* users will have to migrate their old config (if
any), though we could provide a migration script
* yet-another dependency (but we could vendor the
parser, there are small implementations)
* may be harder to write (how does a user know the
expected type of each key? Should [yt]\n loglevel
by a string or a number?)
* for developers: we'd need to change the way the
config is supported in the codebase (currently
we're relying on ConfigParser)
* there are some gotchas, for example boolean true is
written "true" not "True"
3. Switch to yaml or json?
What is your opinion? As a starting point, you could think
about how you'd implement the example given above to
deactivate the log scale for plots made with any field with
type "gas", except pressure (is this something we want? If
so, what's the best solution to achieve it in a meaningful
way for the user?).
Regards,
Corentin
--
Dr. Corentin Cadiou
Post Doctoral Research Assistant
Cosmoparticle Initiative Hub, desk 27
University College London (UCL)
Gower St, Bloomsbury, London WC1E 6BT
mobile: +33.6.43.18.66.83
_______________________________________________
yt-dev mailing list -- yt-dev@python.org
<mailto:yt-dev@python.org>
To unsubscribe send an email to yt-dev-leave@python.org
<mailto:yt-dev-leave@python.org>
https://mail.python.org/mailman3/lists/yt-dev.python.org/
<https://mail.python.org/mailman3/lists/yt-dev.python.org/>
Member address: matthewturk@gmail.com
<mailto:matthewturk@gmail.com>
_______________________________________________
yt-dev mailing list -- yt-dev@python.org <mailto:yt-dev@python.org>
To unsubscribe send an email to yt-dev-leave@python.org
<mailto:yt-dev-leave@python.org>
https://mail.python.org/mailman3/lists/yt-dev.python.org/
<https://mail.python.org/mailman3/lists/yt-dev.python.org/>
Member address: brittonsmith@gmail.com
<mailto:brittonsmith@gmail.com>
yt-dev mailing list -- yt-dev@python.org To unsubscribe send an email to yt-dev-leave@python.org https://mail.python.org/mailman3/lists/yt-dev.python.org/
Member address: corentin.cadiou@iap.fr
Dr. Corentin Cadiou Post Doctoral Research Assistant Cosmoparticle Initiative Hub, desk 27 University College London (UCL) Gower St, Bloomsbury, London WC1E 6BT
mobile: +33.6.43.18.66.83
<OpenPGP_signature.sig>
yt-dev mailing list -- yt-dev@python.org To unsubscribe send an email to yt-dev-leave@python.org https://mail.python.org/mailman3/lists/yt-dev.python.org/ Member address: corentin.cadiou@iap.fr
-- Dr. Corentin Cadiou Post Doctoral Research Assistant Cosmoparticle Initiative Hub, desk 27 University College London (UCL) Gower St, Bloomsbury, London WC1E 6BT
mobile: +33.6.43.18.66.83
On 26 Nov 2020, at 17:44, Corentin CADIOU corentin.cadiou@iap.fr wrote:
We could of course fix the ambiguity by setting priority rules (ex. in this case, we could decide a field name is more specific than a field type, so it should have priority). What do others think about this?
That’s fine by me for reference, I was merely curious, I don’t have a strong opinion on how it should work.
TLDR; to ease the process of others weighing in, we have three options (I'd actually be in favor of Clement's option):
- do not support local config files,
- Clement's proposition: load the local config file (if any), otherwise load the global config file,
- Corentin's proposition: load both the local config file (if any) and the global config file + provide a way to dump the current configviaa CLI argument.
More details:
The way the PR is implementedcurrentlyis that it loads the config from ~/.config/yt/yt.toml, then it walks up the folder tree until it finds a yt.toml file. At the first match (if any), it reads it, overrides whatever was defined earlier and stops walking the tree up (i.e. at most one local yt.toml file will ever be read, in the same spirit as git stopping at the first parent folder that contains a .git directory).
If I am correct, your proposition is the following: if any local config file is found,
then do not read the global config (~/.config/yt/yt.toml). If we were to adopt this
proposition, which file should yt config set foo bar configure?
Here’s a suggestion:
- if no local config file is present already, store the new param in the global file
- else, error out and require that the command be rerun with either
—localor—global
I think this covers every possible use case rather simply and don’t put us in a position where the feature is either hard to implement, document, or understand from the user pov.
-
Britton Smith -
Clément Robert -
Corentin CADIOU -
Matthew Turk