Mailman 3 March 2021 - yt-dev

Important announcements for all developers: yt 4.0 roadmap
by John Zuhone 23 Apr '21

23 Apr '21

Dear yt-dev, The steering committee met last week to determine the operation of the yt project for the next few months until yt 4.0 is released. It is our intention that 4.0 will come out in May or early June 2021. We want to summarize here how--to ensure that 4.0 can come out in this timeline--our process will differ from our classical operations and also what will continue after the release. We are so excited by the new development that has been going on in yt, but as a project we are resource limited and need to make some difficult decisions to ensure the release happens in a timely fashion. Until the yt 4.0 release, the steering committee recommends that project resources (PR triage time, coordinated development effort, reviews, testing support, etc.) be focused on PRs that are directly related to (1) allowing the yt-4.0 release to occur or (2) are bugs that are breaking yt or giving inconsistent/incorrect results. In practice, this means that we are asking you to hold off submitting PRs that are about style or unnecessary improvements to functionality. Otherwise, we are using our triage time to figure out whether a PR is required for the yt-4.0 release rather than diving in and moving our project forward. You are welcome to open any number of issues as placeholders for the work that you want to do rather than opening PRs. Resolving issues will be welcome after the release or during housekeeping week, depending on the content of the issue. A few other items from the steering committee discussion: Housekeeping week We will have a housekeeping week to focus on cleaning up the codebase. This is when we want dead code, code styling, and other PRs both submitted and merged. Housekeeping week will be on June 7-11 this year. PR Mentorship Week This is a week that we will help get older PRs and PRs from new contributors across the finish line. PRs from new contributors are always welcome; this week is specifically for extra focused time to boost the work of new contributors. This will include dedicated time from the development community on taking over PRs or mentoring newer contributors whose PRs have been waiting a while. PR Mentorship week will be held on August 2-6 this year. The pre-commit bot The steering committee has discussed the pre commit bot and finds that the bot actively committing to open PRs with formatting changes on push is having adverse unforeseen complications for developers. We believe the bot represents a barrier for the following reasons: • It fills the git history of larger PRs with lots of formatting commits. It is not reasonable to expect a developer or contributor to interactively rebase to get rid of these. • It requires that developers always pull from their own development branches or risk having to resolve a merge commit in order to continue their development work. • It makes changes without consent to developer and contributor work. Developers and contributors should be able to resolve formatting in their own time and individually if they wish. With this reasoning the steering committee has decided: • The pre-commit.ci bot will be deactivated on the main yt project/yt repository. • We will return to styling checks as part of our CI checks in a PR. If black, isort, or anything else is not appeased, it will fail and cannot be merged. • We will create a bot that can be called on PRs to format styling if it is requested by the PR author or a yt-project member. This can be used if a developer does not feel like performing manual changes. As always, please remember: • All developers are human, and it takes time to submit and review PRs. Be kind in your responses to one another. • Consider the effects of changes in PRs on current developers, on users, and on downstream packages. These are all important parties in the yt-project community. • Do not create PRs that negate another contributor’s active PR. This is demotivating to the other contributor and differences in code opinion should go through the decision making process as outlined in our governance. • Please try to discuss code changes in a public channel before developing on them. • Keep discussions about code in public channels (e.g. stylistic suggestions, differences in implementation, disagreements about code, etc.etc.). This will help maintain transparency of the software development and decision-making process. Thank you all for being such a wonderful and vibrant development community! We are looking forward to the 4.0 release and will see you on the mailing list, slack, github, etc. The yt project steering committee

3 2

New frontend for Parthenon/AthenaPK
by forrestglines＠gmail.com 28 Mar '21

28 Mar '21

Hi yt-dev, I'm working on a yt front end for Parthenon, which is a AMR framework we're developing with Los Alamos based on Athena++ but using the Kokkos performance portability library to run on CPUs and GPUs. AthenaPK is our downstream physics code using Parthenon, currently developed at Michigan State University in collaboration with Jim Stone. So far I have a basic frontend based heavily on the existing Athena++ frontend, although some attributes have been renamed and datasets rearranged in Parthenon. We're still working on the yt frontend and changing our output format to better accommodate `yt`, but at some point we would like our frontend to enter the yt code base. What's the typical threshold for inclusion in yt? Links for Parthenon, AthenaPK, and my WIP frontend https://github.com/lanl/parthenon.git https://gitlab.com/theias/hpc/jmstone/athena-parthenon/athenapk https://github.com/forrestglines/yt/tree/athenapk-frontend Thanks Forrest

2 1

Master to main
by Clément Robert 23 Mar '21

23 Mar '21

Hi yt-dev The yt repo’s master branch was renamed to “main” a few weeks back and it seems that it went mostly okay save a few bumps in the first day. Now I’m wondering, for the sake of uniformity, would it be ok to apply the same change to the other repos in the project as well ? Best Clément

3 2

Questions regarding "per-field config"
by Corentin CADIOU 23 Mar '21

23 Mar '21

Hi all, My PR to add support for per-field configuration (https://github.com/yt-project/yt/pull/1931) is now ready code-wise. The only question remaining is the exact format of the configuration file. The PR adds support for configuring the plot behaviour of individual fields, allowing to set the default colormap, units and log scale of any field. Currently, this is stored in the config file as [gas.density] # this will set the density plotting options cmap = "plasma" log = true units = "mp/cm**3" [gas.velocity_divergence] cmap = "bwr" log = false [deposit] path_length_units = "kpc" # use kpc for projections instead of cm but the plot config could be stored in a section of its own, like � [ANYTHING.gas.density] cmap = "plasma" log = true units = "mp/cm**3" [ANYTHING.gas.velocity_divergence] cmap = "bwr" log = false [ANYTHING.deposit] path_length_units = "kpc" Where we could substitute ANYTHING with e.g. plots, plotting_configuration or please_let_me_handcraft_my_plot. What are your opinion on how to keep this config file tidy? Best, Corentin -- Dr. Corentin Cadiou Post Doctoral Research Assistant Cosmoparticle Initiative Hub, desk 27 University College London (UCL) Gower St, Bloomsbury, London WC1E 6BT mobile: +33.6.43.18.66.83

3 2

"churn week" doodle poll--Fill out by March 17!
by Stephanie Tonnesen 17 Mar '21

17 Mar '21

Dear yt-dev, As you are all well aware, yt is a constantly growing project with lots of possible improvements that could update the codebase--indeed, many of the pull requests currently waiting to be merged are small or large improvements to yt! However, some updates to the code have a lot of strings attached and so could use some more concerted effort by a few developers to get them fully merged into yt. As such, we would like to try and schedule 1-2 weeks this summer as "churn weeks": making yt buttery! We want as many people as are interested to be involved, where involvement loosely looks like 2-3 hours/day where you can really try and buckle down to work through pull requests and code updates with others via slack or whatever works best for you. So, if you might be interested in this, please fill out the doodle for weeks that may work this summer! "Yes" = 4-5 days in which you should be able to give 2-3 hours to churning yt, "If need be" = 3 days in which you have 2-3 hours of churning time, "No" = less than that. https://www.doodle.com/poll/n3zr4mwwxmap64n7?utm_source=poll&utm_medium=link Thanks everyone!! Best, Stephanie -- Dr. Stephanie Tonnesen Associate Research Scientist CCA, Flatiron Institute New York, NY stonnes(a)gmail.com

1 1

HaloCatalog PR
by Britton Smith 12 Mar '21

12 Mar '21

Hi everyone, I just finished off a PR I started quite a while ago to make some substantial improvements to the HaloCatalog in the yt_astro_analysis package. The major updates are: - support for halo finding on a time-series for all halo finders (fof, hop, and rockstar). You no longer have to run the RockstarHaloFinder separately or call HaloCatalog in a loop for hop/fof. - the loop over all halos in HaloCatalog now happens one io chunk at a time, instead of just querying ds.all_data(). This happens only on the root process now, significantly reducing io and time to being iterating. - vastly updated halo finding documentation. If anyone would like to offer comments on this, please find the pull request at the link below: https://github.com/yt-project/yt_astro_analysis/pull/62 Thanks! Britton

1 0

Naming convention changes for energy and momentum-related fields: may affect your frontend
by John ZuHone 11 Mar '21

11 Mar '21

Hi folks, I’ve written a PR which (among other things) defines some new standards for yt-4.0 in terms of naming fields which represent energy and momentum-related quantities. In short, the issue is this: the various energy fields in yt have an inconsistent naming scheme. For example, the "kinetic_energy" derived field is an energy density (say ergs/cm^3), but "thermal_energy", which usually is aliased from output fields in frontends, is usually a specific energy (erg/g). Also, momentum fields, which they appear in certain frontends, are almost always "momentum density" fields instead. The PR addresses this inconsistency by renaming fields wherever they appear using the following scheme: Energy density fields are named like this: "kinetic_energy" --> "kinetic_energy_density" Specific energy fields are named like this: "thermal_energy" --> "specific_thermal_energy" Momentum density fields are named like this: "momentum_x" --> "momentum_density_x" Eventually, we really ought to have some kind of domain context which handles all this--and this PR should be seen as the prep work for something like that, in addition to a new class which handles different equations of state which I would also like to do. So why should you care? One reason may be that this PR affects field definitions for energy and/or momentum fields in a number of frontends. They are ART ARTIO Athena Athena++ Boxlib Chombo Enzo Enzo-P FLASH GAMER GDF GIZMO SPH Stream If you are a maintainer or a user of one of these frontends, it may be a good idea to check my proposed changes to see if they are in the spirit of what you and other users would be fine with. In any case, the PR is here for your review: https://github.com/yt-project/yt/pull/3059 Comments, suggestions welcome. Best, John

2 2

YTEP-0039: rich Terminal User Interface
by Clément Robert 07 Mar '21

07 Mar '21

Hi yt-dev Here’s a little YTEP I’ve been working on, in which I propose modernising and improving our terminal output (mostly logs) looks using rich (https://github.com/willmcgugan/rich). https://github.com/yt-project/ytep/pull/18 The logging part is completely implemented in https://github.com/yt-project/yt/pull/3106 Both PRs are up for comments, questions and reviews. Take care Clément

2 2

Nominate Chris Havlin
by Clément Robert 05 Mar '21

05 Mar '21

Hi yt-dev ! The idea came up twice today (once at the triage meeting) so I would like to nominate Chris Havlin for membership. Chris has been consistently active in the community, participating to meetings and the mailing list, and he has already took on the role of a helpful reviewer. What do you guys think ? Talk to you soon Clément

4 4

Dask + yt development plan
by Chris Havlin 04 Mar '21

04 Mar '21

Hello! I'm writing to follow up on the discussion of a Dask feature branch at last week's development meeting. I wanted to summarize the plan to move forward with a Dask feature branch, add some additional notes and make sure everyone has a chance for initial feedback before we move forward with creating a feature branch. Generally, the plan is to: 1. create a feature branch named `dask` 2a. submit dask-specific PRs to `dask` feature branch 2b. submit general PRs that come up in dask development to `main` 3. weekly (at a minimum, more frequent cadence is welcome) merges of `main` into `dask` branch The use of a feature branch will allow PR review throughout development, avoiding a massive review when finally merging the `dask` branch back into `main`. It also allows multiple developers to work on Dask development more easily. Some extra notes and clarification: Dask-specific vs general PRs: Some changes to add dask support may be better as PRs to `main`. These changes should be non-breaking and not rely on Dask itself. For example, PR 2934 (https://github.com/yt-project/yt/pull/2934 ) added pickle support to some selection objects to help with Dask development but it has general applicability so was submitted to `main`. If it's not clear whether a PR is general enough, it should be submitted to the `dask` branch and reviewers can suggest re-targeting to `main` if general enough. Feature branch name: I'm using `dask` everywhere here… but it could be more exciting. `thedaskening` perhaps (credit to Madicken for this name!)? Feedback on feature branch name is encouraged :) Dependencies: Current development makes Dask a hard dependency, but a full Dask install is not required. At present the minimal install requires the `array`, `distributed` and `delayed` dependency sets. e.g., python -m pip install "dask[array,delayed,distributed]" This is not far from a full Dask install, so it may be simpler to just require `dask[complete]`. The complete install adds the dask dataframe, dask bag and dask diagnostic features. The extra dependencies that these subsets include are pandas, ffspec, toolz and bokeh (see https://github.com/dask/dask/blob/master/setup.py). Of those, perhaps bokeh is extraneous enough that we should only require the minimal install (bokeh is only used for Dask's interactive browser-dashboard for monitoring Client/cluster activity). YTEP: planning to start drafting a YTEP after a feature branch is started and some development has proceeded and we have some broader input. Short Term Development Targets: A couple of short term work directions I've been planning include: 1. a daskified particle reader, currently in my fork of yt here: https://github.com/chrishavlin/yt/tree/dask_init_particle 2. Daskification of derived quantities and "simpler" chunked operations Please reply with any comments you may have! I'm excited about getting feedback and moving this work forward! Cheers, Chris

4 4