Mailman 3 October 2021 - yt-dev

Fwd: Supporting arbitrary grid schemes ?
by Clément Robert Nov. 24, 2021

Nov. 24, 2021

Sorry I accidentally responded to Micheal only, here’s my response > Begin forwarded message: > > From: Clément Robert <clement.robert(a)protonmail.com> > Subject: Re: [yt-dev] Supporting arbitrary grid schemes ? > > Date: 20 October 2021 at 17:04:12 CEST > To: Michael Zingale <michael.zingale(a)stonybrook.edu> > > Interesting. I think yt already has some support for these, though I admit I’ve never seen it used outside of (failed) image tests, so here’s one I was able to find on my computer. > Does this fall into the “mapped grid” catagory ? > > Indeed what I want to support in the context of my current team is probably a subset of these “logically rectangular” grids. My teams uses rectilinear grids, but we stil have orthogonality everywhere. > >> On 20 Oct 2021, at 15:41, Michael Zingale <michael.zingale(a)stonybrook.edu> wrote: >> >> here's an example of some simple mapped grids that are used in astro: >> >> http://faculty.washington.edu/rjl/pubs/circles/SIR000723.pdf >> >> it sounds like what you are describing is a restricted version of a general mapped grid. >> >> On Wed, Oct 20, 2021 at 9:31 AM Clément Robert via yt-dev <yt-dev(a)python.org> wrote: >> >>> I don’t know ! Where can I find a definition for these ? >>> >>>> On 20 Oct 2021, at 13:52, Michael Zingale <michael.zingale(a)stonybrook.edu> wrote: >>>> >>>> It's this similar to mapped grids? >>>> >>>> On Wed, Oct 20, 2021, 5:33 AM Clément Robert via yt-dev <yt-dev(a)python.org> wrote: >>>> >>>>> Hi folks >>>>> >>>>> As you may be aware, yt’s grid based index classes offer support for patched-based AMR as well octrees, but it’s currently lacking support for so called “stretched gridding scheme”. This is a known limitation in several existing frontends (AMRVAC, BoxLib, Chombo-Pluto are the ones I know of). >>>>> >>>>> For context, what I refer to as “stretched grids” are grids where cell sizes are not simple powers of 1/2 (with respect to a unitary reference) as in classic AMR and octrees, but may be specified by arbitrary rules (such as geometric series). Some hydro code support having parts of the grid following classic AMR scheme and other blocks to be “stretched” along one or several directions at once. This is the case of AMRVAC and Pluto, for instance. BoxLib also has support for complex refinement rules that are not covered by storing a single `refine_by` integer as we’re currently doing in Index classes (this is https://github.com/yt-project/yt/issues/3481). >>>>> It seems to me that the “easy” way of supporting these data formats would be to generalise yt’s mechanisms for reconstructing cell position and size (currently it’s using cell index, domain edges, refinement level + a `refine_by` multiplier) so that cell edges can be stored as arrays that frontends would need to somehow retrieve from datafile or reconstruct from parameter files. >>>>> >>>>> There’s been some effort in the past to get over this limitation led by John and Matt, and we’ve been hitting this wall from time to time in discussions. >>>>> Now, I understand that this would most likely not be a trivial change, especially because we don’t want to break existing frontends in the process (or performance !), and at the moment, I don’t have a very clear idea of what exact steps are needed to achieve this goal, and in particular I don’t have enough material to draft a YTEP on the topic. >>>>> It seems clear we’d need *some* refactoring/generalisation of Index classes and friend classes and new pixelizer routines would likely be needed. Anything else ? >>>>> >>>>> I however have personal motivation to move forward here since my team is developing an hydro code and running simulations where grid streching is the rule rather than the exception. >>>>> >>>>> What I would like to do to get us rolling is simply collect the existing issues and PRs related to this goal. I know there are a couple, but it’s very easy to loose track of them, and it’s hard to see the big picture. >>>>> Can I create a label (“arbitrary grids” ?) and apply it to existing material touching this topic ? >>>>> Do you know of existing PRs and issues that are relevant to this topic ? >>>>> >>>>> Here are the ones I know: >>>>> - https://github.com/yt-project/yt/issues/3481 >>>>> - https://github.com/yt-project/yt/pull/2998 >>>>> - https://github.com/yt-project/yt/issues/1880 >>>>> - https://github.com/yt-project/yt/issues/385 (from 2012 !!) >>>>> >>>>> Thanks >>>>> Clément >>>>> _______________________________________________ >>>>> yt-dev mailing list -- yt-dev(a)python.org >>>>> To unsubscribe send an email to yt-dev-leave(a)python.org >>>>> https://mail.python.org/mailman3/lists/yt-dev.python.org/ >>>>> Member address: michael.zingale(a)stonybrook.edu >>> >>> _______________________________________________ >>> yt-dev mailing list -- yt-dev(a)python.org >>> To unsubscribe send an email to yt-dev-leave(a)python.org >>> https://mail.python.org/mailman3/lists/yt-dev.python.org/ >>> Member address: michael.zingale(a)stonybrook.edu >> >> -- >> >> Michael Zingale >> Professor of Physics and Astronomy >> >> Dept. of Physics & Astronomy • Stony Brook University • Stony Brook, NY 11794-3800 >> phone: 631-632-8225 >> e-mail: michael.zingale(a)stonybrook.edu >> web: [https://zingale.github.io](https://zingale.github.io/) >> github: [https://github.com/zingale](http://github.com/zingale)

4 7

Removing deprecated code on the main branch
by Clément Robert Oct. 24, 2021

Oct. 24, 2021

Hi folks, A bunch of functions have been deprecated throughout yt’s life, and in particular, many of these were explicitly targeted for removal post 4.0 The following PR cleans up most of this material https://github.com/yt-project/yt/pull/3240 I do not expect this to be controversial since most of the removed content was deprecated long before yt 4.0 when we didn’t have a clear deprecation cycle, but just in case, I’d like to advertise the PR for discussion. If anyone is currently developing stuff that depends directly on the main branch and needs some additional period of time to adjust to these deprecations, let’s hear them. Here’s a couple relevant remarks: - downstream projects should be able to detect deprecation warnings automatically by configuring CI to treat warnings as errors, which is something we’ve been doing with yt itself lately. - we’ve deprecated usage of ambiguous fields (e.g., using ‘density’ instead of (“gas”, “density”) when (“stream”, “density”) also exists), but this part is intentionally left out of the PR because it is not trivial to deal with internally. - this work will *not* affect the yt 4.0.x series, so it’ll stay possible to use the deprecated stuff with versions released up to now (and any future bug fix release in the 4.0 series) So in short: any objections to merging this asap ? Cheers, Clément

1 0

Dropping Python 3.6
by Clément Robert Oct. 22, 2021

Oct. 22, 2021

Dear yt dev team, As was previously mentioned in yt 4.0.0's release notes, we're currently planning to drop support for Python 3.6 in our next feature release (4.1.0) I propose that this change be performed on the dev branch of yt by late October this year, but not earlier than October the 20th. As was discussed on Slack, I would like to send an official announcement to yt-users(a)python.org, but I’ll leave open to discussion for a week first, or as long as necessary for the discussion to resolve happily. — Here's the PR to drop Python 3.6 https://github.com/yt-project/yt/pull/2917 Please take minute to either request changes, discuss and/or approve it. — Nice implications This means we'll soon be able to use Python 3.7+ only syntax and features. Here are the two key highlights that I personally think are most impactful for yt: - `from __future__ import annotations` can be used to leverage modern Python annotations features from more recent versions (as late as 3.11 !) This will be extremely useful to improve typing in yt, and in support to YTEP 0038 https://github.com/yt-project/ytep/pull/17 Here's a gist with some showcase examples of how this __future__ import helps simplifying typed code https://gist.github.com/neutrinoceros/0eaa046a00689254dee0738c65c9bdfc - the `dataclasses` module becomes available https://docs.python.org/3/whatsnew/3.7.html#whatsnew37-pep557 See the detailed "what's new" note from Python 3.7.0 for more https://docs.python.org/3/whatsnew/3.7.html Additional notes - Python 3.6 is 5 years old and will be EOL (end of life) by December https://www.python.org/dev/peps/pep-0494/ - Numpy dropped support for Python 3.6 in numpy 1.20.0, released in January 2021 https://github.com/numpy/numpy/releases/tag/v1.20.0 - Matplotlib dropped support fot Python 3.6 in matplotlib 3.4.0, released in March 2021 https://github.com/matplotlib/matplotlib/commit/f9907b49f9d00c03cf014a25d05… - Python 3.10 will be officially released early October https://www.python.org/dev/peps/pep-0619/ - Adding support for 3.10 should be a breeze^{TM}: we've been running a subset of our test suite against it (+ future versions of matplotlib and numpy) for a couple months, so the expected most important bottleneck for us will depend on when numpy and matplotlib are able to publish wheels for 3.10 - The plan is to first add support to 3.10 and *then* drop 3.6. - This is in agreement with NEP 29, which was coauthored by our own release manager, Madicken Munk https://numpy.org/neps/nep-0029-deprecation_policy.html - Various 3.7+ idioms will now be continuously enforced with pyupgrade https://github.com/asottile/pyupgrade which is already part of our pre-commit toolbelt. Cheers, and have a great week Clément

6 9

Supporting arbitrary grid schemes ?
by Clément Robert Oct. 20, 2021

Oct. 20, 2021

Hi folks As you may be aware, yt’s grid based index classes offer support for patched-based AMR as well octrees, but it’s currently lacking support for so called “stretched gridding scheme”. This is a known limitation in several existing frontends (AMRVAC, BoxLib, Chombo-Pluto are the ones I know of). For context, what I refer to as “stretched grids” are grids where cell sizes are not simple powers of 1/2 (with respect to a unitary reference) as in classic AMR and octrees, but may be specified by arbitrary rules (such as geometric series). Some hydro code support having parts of the grid following classic AMR scheme and other blocks to be “stretched” along one or several directions at once. This is the case of AMRVAC and Pluto, for instance. BoxLib also has support for complex refinement rules that are not covered by storing a single `refine_by` integer as we’re currently doing in Index classes (this is https://github.com/yt-project/yt/issues/3481). It seems to me that the “easy” way of supporting these data formats would be to generalise yt’s mechanisms for reconstructing cell position and size (currently it’s using cell index, domain edges, refinement level + a `refine_by` multiplier) so that cell edges can be stored as arrays that frontends would need to somehow retrieve from datafile or reconstruct from parameter files. There’s been some effort in the past to get over this limitation led by John and Matt, and we’ve been hitting this wall from time to time in discussions. Now, I understand that this would most likely not be a trivial change, especially because we don’t want to break existing frontends in the process (or performance !), and at the moment, I don’t have a very clear idea of what exact steps are needed to achieve this goal, and in particular I don’t have enough material to draft a YTEP on the topic. It seems clear we’d need *some* refactoring/generalisation of Index classes and friend classes and new pixelizer routines would likely be needed. Anything else ? I however have personal motivation to move forward here since my team is developing an hydro code and running simulations where grid streching is the rule rather than the exception. What I would like to do to get us rolling is simply collect the existing issues and PRs related to this goal. I know there are a couple, but it’s very easy to loose track of them, and it’s hard to see the big picture. Can I create a label (“arbitrary grids” ?) and apply it to existing material touching this topic ? Do you know of existing PRs and issues that are relevant to this topic ? Here are the ones I know: - https://github.com/yt-project/yt/issues/3481 - https://github.com/yt-project/yt/pull/2998 - https://github.com/yt-project/yt/issues/1880 - https://github.com/yt-project/yt/issues/385 (from 2012 !!) Thanks Clément

2 3

backport system
by Matthew Turk Oct. 18, 2021

Oct. 18, 2021

Hi folks, I made an issue about using a backport service for backporting bugfixes. I would like to request comments on it go to that issue, and I will summarize here if we reach a conclusion. https://github.com/yt-project/yt/issues/3553 -Matt

3 6

A New bugfix release (4.0.2) ?
by Clément Robert Oct. 12, 2021

Oct. 12, 2021

Hi yt-dev ! I hope you’re all doing well. I would like to propose we plan for another bug fix release for the yt 4.0.x series. We had approximately 20 merged bug fixes since yt 4.0.1 (excluding docs and infrastructure fixes), and a couple bug fixing PRs that could realistically go in soon. So how about we start a 4.0.2 milestone to keep track of our progress with backports ? This time around, they’ll have to be manually performed, which I believe is one reason why we don’t do this more often, but in the future, we could automate the back porting process. For instance, Matplotlib and Ipython are using this tool: https://meeseeksbox.github.io As far as I understand, this bot can create backport PRs mirroring patches associated with a milestone, itself attached to dedicated release branch. Let me know if you have any thoughts on that too. Cheers, Clément

4 8

Typing pull request
by Matthew Turk Oct. 11, 2021

Oct. 11, 2021

Hi folks, Clement has prepared a rather large and expansive pull request that adds type hinting to the current yt codebase. I have read and reviewed it, and I find it to be an improvement over our current situation, *without* causing any backward-incompatible issues. https://github.com/yt-project/yt/pull/3546 It adds a few things that I'd like to highlight: 1. mypy checking, which someone else can explain better than I can, but which checks that we are in fact passing the types to different functions that we think we are. 2. better support for hinting at variable types in IDEs such a VS Code, to reduce the likelihood of errors 3. a system for updating our usage of type hints in a forward compatible way I would like to propose that once it has been marked as non-draft, we prepare ourselves to merge it. While adding type hinting to a codebase can be subject to a number of debates -- which I have every intention of participating in! -- I would like to propose that we try to prepare ourselves to lower the barrier for this particular set of changes, to avoid being put in a situation where everyone agrees on the end result but the latency caused by decision making around specifics causes the decision to languish and continue indefinitely. So I'd ask that folks that are interested in the *first* (but absolutely *not final*) pass at adding type hints examine the pull request and provide feedback, and we move forward with it expeditiously. (For instance, as folks have seen in the "particle reader cleanup" pull request, I have a difference of opinion with Clement about some specifics -- but I do not want that to get in the way of the first iteration being sealed off and brought into the code base.) -Matt

5 9

Some updates to PR workflow
by Matthew Turk Oct. 9, 2021

Oct. 9, 2021

Hi all, I've been tidying up the PRs a bit. I wanted to share a couple things: 1. I've added branch protections. This means that, except for administrators (we can also make administrators do it) it now mandates that these status checks be satisfied before merging: - "Mergeable" - "WIP" - "py38" (This is not the full set -- it notably excludes OS specific builds and whatnot. But this will catch almost all the main things we reject PRs for. It would also be possible to add *everything* to this list, but I wasn't sure if that was warranted.) 2. All pull requests have to come from a branch. We mostly mandate this already. 3. All PRs require at least one approval. (There are more fine-grained protections we could add if we were to develop on a branch other than main.) This now lets us use what's called "auto-merge," This means that on any PR that does not have these *yet* met (specifically, status checks) we can say, "if this passes, merge it." This should improve the efficiency of large PR triages, as it means that we can review, queue things for merging and then it will merge *if-and-only-if* those tests pass. I'm going to propose a couple other minor-ish changes to help streamline the workflow, especially for changelogs, but the main ones there are that I'm going to propose (once I've written it up) changing the labels a little bit and also freely editing issue titles. -Matt

2 2