The steering committee met last week to determine the operation of the yt project for the next few months until yt 4.0 is released. It is our intention that 4.0 will come out in May or early June 2021. We want to summarize here how--to ensure that 4.0 can come out in this timeline--our process will differ from our classical operations and also what will continue after the release. We are so excited by the new development that has been going on in yt, but as a project we are resource limited and need to make some difficult decisions to ensure the release happens in a timely fashion.
Until the yt 4.0 release, the steering committee recommends that project resources (PR triage time, coordinated development effort, reviews, testing support, etc.) be focused on PRs that are directly related to (1) allowing the yt-4.0 release to occur or (2) are bugs that are breaking yt or giving inconsistent/incorrect results. In practice, this means that we are asking you to hold off submitting PRs that are about style or unnecessary improvements to functionality. Otherwise, we are using our triage time to figure out whether a PR is required for the yt-4.0 release rather than diving in and moving our project forward. You are welcome to open any number of issues as placeholders for the work that you want to do rather than opening PRs. Resolving issues will be welcome after the release or during housekeeping week, depending on the content of the issue.
A few other items from the steering committee discussion:
We will have a housekeeping week to focus on cleaning up the codebase. This is when we want dead code, code styling, and other PRs both submitted and merged. Housekeeping week will be on June 7-11 this year.
PR Mentorship Week
This is a week that we will help get older PRs and PRs from new contributors across the finish line. PRs from new contributors are always welcome; this week is specifically for extra focused time to boost the work of new contributors. This will include dedicated time from the development community on taking over PRs or mentoring newer contributors whose PRs have been waiting a while. PR Mentorship week will be held on August 2-6 this year.
The pre-commit bot
The steering committee has discussed the pre commit bot and finds that the bot actively committing to open PRs with formatting changes on push is having adverse unforeseen complications for developers. We believe the bot represents a barrier for the following reasons:
• It fills the git history of larger PRs with lots of formatting commits. It is not reasonable to expect a developer or contributor to interactively rebase to get rid of these.
• It requires that developers always pull from their own development branches or risk having to resolve a merge commit in order to continue their development work.
• It makes changes without consent to developer and contributor work. Developers and contributors should be able to resolve formatting in their own time and individually if they wish.
With this reasoning the steering committee has decided:
• The pre-commit.ci bot will be deactivated on the main yt project/yt repository.
• We will return to styling checks as part of our CI checks in a PR. If black, isort, or anything else is not appeased, it will fail and cannot be merged.
• We will create a bot that can be called on PRs to format styling if it is requested by the PR author or a yt-project member. This can be used if a developer does not feel like performing manual changes.
As always, please remember:
• All developers are human, and it takes time to submit and review PRs. Be kind in your responses to one another.
• Consider the effects of changes in PRs on current developers, on users, and on downstream packages. These are all important parties in the yt-project community.
• Do not create PRs that negate another contributor’s active PR. This is demotivating to the other contributor and differences in code opinion should go through the decision making process as outlined in our governance.
• Please try to discuss code changes in a public channel before developing on them.
• Keep discussions about code in public channels (e.g. stylistic suggestions, differences in implementation, disagreements about code, etc.etc.). This will help maintain transparency of the software development and decision-making process.
Thank you all for being such a wonderful and vibrant development community! We are looking forward to the 4.0 release and will see you on the mailing list, slack, github, etc.
The yt project steering committee
I'm working on a yt front end for Parthenon, which is a AMR framework we're developing with Los Alamos based on Athena++ but using the Kokkos performance portability library to run on CPUs and GPUs. AthenaPK is our downstream physics code using Parthenon, currently developed at Michigan State University in collaboration with Jim Stone. So far I have a basic frontend based heavily on the existing Athena++ frontend, although some attributes have been renamed and datasets rearranged in Parthenon. We're still working on the yt frontend and changing our output format to better accommodate `yt`, but at some point we would like our frontend to enter the yt code base. What's the typical threshold for inclusion in yt?
Links for Parthenon, AthenaPK, and my WIP frontend
The yt repo’s master branch was renamed to “main” a few weeks back and it seems that it went mostly okay save a few bumps in the first day.
Now I’m wondering, for the sake of uniformity, would it be ok to apply the same change to the other repos in the project as well ?
My PR to add support for per-field configuration
(https://github.com/yt-project/yt/pull/1931) is now ready code-wise. The
only question remaining is the exact format of the configuration file.
The PR adds support for configuring the plot behaviour of individual
fields, allowing to set the default colormap, units and log scale of any
field. Currently, this is stored in the config file as
[gas.density] # this will set the density plotting options
cmap = "plasma"
log = true
units = "mp/cm**3"
cmap = "bwr"
log = false
path_length_units = "kpc" # use kpc for projections instead of cm
but the plot config could be stored in a section of its own, like
cmap = "plasma"
log = true
units = "mp/cm**3"
cmap = "bwr"
log = false
path_length_units = "kpc"
Where we could substitute ANYTHING with e.g. plots,
plotting_configuration or please_let_me_handcraft_my_plot.
What are your opinion on how to keep this config file tidy?
Dr. Corentin Cadiou
Post Doctoral Research Assistant
Cosmoparticle Initiative Hub, desk 27
University College London (UCL)
Gower St, Bloomsbury, London WC1E 6BT
As you are all well aware, yt is a constantly growing project with lots of
possible improvements that could update the codebase--indeed, many of the
pull requests currently waiting to be merged are small or large
improvements to yt!
However, some updates to the code have a lot of strings attached and so
could use some more concerted effort by a few developers to get them fully
merged into yt.
As such, we would like to try and schedule 1-2 weeks this summer as "churn
weeks": making yt buttery! We want as many people as are interested to be
involved, where involvement loosely looks like 2-3 hours/day where you can
really try and buckle down to work through pull requests and code updates
with others via slack or whatever works best for you.
So, if you might be interested in this, please fill out the doodle for
weeks that may work this summer! "Yes" = 4-5 days in which you should be
able to give 2-3 hours to churning yt, "If need be" = 3 days in which you
have 2-3 hours of churning time, "No" = less than that.
Dr. Stephanie Tonnesen
Associate Research Scientist
CCA, Flatiron Institute
New York, NY
I just finished off a PR I started quite a while ago to make some
substantial improvements to the HaloCatalog in the yt_astro_analysis
package. The major updates are:
- support for halo finding on a time-series for all halo finders (fof,
hop, and rockstar). You no longer have to run the RockstarHaloFinder
separately or call HaloCatalog in a loop for hop/fof.
- the loop over all halos in HaloCatalog now happens one io chunk at a
time, instead of just querying ds.all_data(). This happens only on
the root process now, significantly reducing io and time to being
- vastly updated halo finding documentation.
If anyone would like to offer comments on this, please find the pull
request at the link below:
I’ve written a PR which (among other things) defines some new standards for yt-4.0 in terms of naming fields which represent energy and momentum-related quantities.
In short, the issue is this: the various energy fields in yt have an inconsistent naming scheme. For example, the "kinetic_energy" derived field is an energy density (say ergs/cm^3), but "thermal_energy", which usually is aliased from output fields in frontends, is usually a specific energy (erg/g). Also, momentum fields, which they appear in certain frontends, are almost always "momentum density" fields instead.
The PR addresses this inconsistency by renaming fields wherever they appear using the following scheme:
Energy density fields are named like this: "kinetic_energy" --> "kinetic_energy_density"
Specific energy fields are named like this: "thermal_energy" --> "specific_thermal_energy"
Momentum density fields are named like this: "momentum_x" --> "momentum_density_x"
Eventually, we really ought to have some kind of domain context which handles all this--and this PR should be seen as the prep work for something like that, in addition to a new class which handles different equations of state which I would also like to do.
So why should you care? One reason may be that this PR affects field definitions for energy and/or momentum fields in a number of frontends. They are
If you are a maintainer or a user of one of these frontends, it may be a good idea to check my proposed changes to see if they are in the spirit of what you and other users would be fine with.
In any case, the PR is here for your review:
Comments, suggestions welcome.
Hi yt-dev !
The idea came up twice today (once at the triage meeting) so I would like to nominate Chris Havlin for membership.
Chris has been consistently active in the community, participating to meetings and the mailing list, and he has already took on the role of a helpful reviewer.
What do you guys think ?
Talk to you soon
I'm writing to follow up on the discussion of a Dask feature branch at last week's development meeting. I wanted to summarize the plan to move forward with a Dask feature branch, add some additional notes and make sure everyone has a chance for initial feedback before we move forward with creating a feature branch.
Generally, the plan is to:
1. create a feature branch named `dask`
2a. submit dask-specific PRs to `dask` feature branch
2b. submit general PRs that come up in dask development to `main`
3. weekly (at a minimum, more frequent cadence is welcome) merges of `main` into `dask` branch
The use of a feature branch will allow PR review throughout development, avoiding a massive review when finally merging the `dask` branch back into `main`. It also allows multiple developers to work on Dask development more easily.
Some extra notes and clarification:
Dask-specific vs general PRs:
Some changes to add dask support may be better as PRs to `main`. These changes should be non-breaking and not rely on Dask itself. For example, PR 2934 (https://github.com/yt-project/yt/pull/2934 ) added pickle support to some selection objects to help with Dask development but it has general applicability so was submitted to `main`. If it's not clear whether a PR is general enough, it should be submitted to the `dask` branch and reviewers can suggest re-targeting to `main` if general enough.
Feature branch name:
I'm using `dask` everywhere here… but it could be more exciting. `thedaskening` perhaps (credit to Madicken for this name!)? Feedback on feature branch name is encouraged :)
Current development makes Dask a hard dependency, but a full Dask install is not required. At present the minimal install requires the `array`, `distributed` and `delayed` dependency sets. e.g.,
python -m pip install "dask[array,delayed,distributed]"
This is not far from a full Dask install, so it may be simpler to just require `dask[complete]`. The complete install adds the dask dataframe, dask bag and dask diagnostic features. The extra dependencies that these subsets include are pandas, ffspec, toolz and bokeh (see https://github.com/dask/dask/blob/master/setup.py). Of those, perhaps bokeh is extraneous enough that we should only require the minimal install (bokeh is only used for Dask's interactive browser-dashboard for monitoring Client/cluster activity).
planning to start drafting a YTEP after a feature branch is started and some development has proceeded and we have some broader input.
Short Term Development Targets:
A couple of short term work directions I've been planning include:
1. a daskified particle reader, currently in my fork of yt here: https://github.com/chrishavlin/yt/tree/dask_init_particle
2. Daskification of derived quantities and "simpler" chunked operations
Please reply with any comments you may have! I'm excited about getting feedback and moving this work forward!