Hi folks,
A bunch of functions have been deprecated throughout yt’s life, and in particular, many of these were explicitly targeted for removal post 4.0
The following PR cleans up most of this material https://github.com/yt-project/yt/pull/3240
I do not expect this to be controversial since most of the removed content was deprecated long before yt 4.0 when we didn’t have a clear deprecation cycle, but just in case, I’d like to advertise the PR for discussion.
If anyone is currently developing stuff that depends directly on the main branch and needs some additional period of time to adjust to these deprecations, let’s hear them.
Here’s a couple relevant remarks:
- downstream projects should be able to detect deprecation warnings automatically by configuring CI to treat warnings as errors, which is something we’ve been doing with yt itself lately.
- we’ve deprecated usage of ambiguous fields (e.g., using ‘density’ instead of (“gas”, “density”) when (“stream”, “density”) also exists), but this part is intentionally left out of the PR because it is not trivial to deal with internally.
- this work will *not* affect the yt 4.0.x series, so it’ll stay possible to use the deprecated stuff with versions released up to now (and any future bug fix release in the 4.0 series)
So in short: any objections to merging this asap ?
Cheers,
Clément
Sorry I accidentally responded to Micheal only, here’s my response
> Begin forwarded message:
>
> From: Clément Robert <clement.robert(a)protonmail.com>
> Subject: Re: [yt-dev] Supporting arbitrary grid schemes ?
>
> Date: 20 October 2021 at 17:04:12 CEST
> To: Michael Zingale <michael.zingale(a)stonybrook.edu>
>
> Interesting. I think yt already has some support for these, though I admit I’ve never seen it used outside of (failed) image tests, so here’s one I was able to find on my computer.
> Does this fall into the “mapped grid” catagory ?
>
> Indeed what I want to support in the context of my current team is probably a subset of these “logically rectangular” grids. My teams uses rectilinear grids, but we stil have orthogonality everywhere.
>
>> On 20 Oct 2021, at 15:41, Michael Zingale <michael.zingale(a)stonybrook.edu> wrote:
>>
>> here's an example of some simple mapped grids that are used in astro:
>>
>> http://faculty.washington.edu/rjl/pubs/circles/SIR000723.pdf
>>
>> it sounds like what you are describing is a restricted version of a general mapped grid.
>>
>> On Wed, Oct 20, 2021 at 9:31 AM Clément Robert via yt-dev <yt-dev(a)python.org> wrote:
>>
>>> I don’t know ! Where can I find a definition for these ?
>>>
>>>> On 20 Oct 2021, at 13:52, Michael Zingale <michael.zingale(a)stonybrook.edu> wrote:
>>>>
>>>> It's this similar to mapped grids?
>>>>
>>>> On Wed, Oct 20, 2021, 5:33 AM Clément Robert via yt-dev <yt-dev(a)python.org> wrote:
>>>>
>>>>> Hi folks
>>>>>
>>>>> As you may be aware, yt’s grid based index classes offer support for patched-based AMR as well octrees, but it’s currently lacking support for so called “stretched gridding scheme”. This is a known limitation in several existing frontends (AMRVAC, BoxLib, Chombo-Pluto are the ones I know of).
>>>>>
>>>>> For context, what I refer to as “stretched grids” are grids where cell sizes are not simple powers of 1/2 (with respect to a unitary reference) as in classic AMR and octrees, but may be specified by arbitrary rules (such as geometric series). Some hydro code support having parts of the grid following classic AMR scheme and other blocks to be “stretched” along one or several directions at once. This is the case of AMRVAC and Pluto, for instance. BoxLib also has support for complex refinement rules that are not covered by storing a single `refine_by` integer as we’re currently doing in Index classes (this is https://github.com/yt-project/yt/issues/3481).
>>>>> It seems to me that the “easy” way of supporting these data formats would be to generalise yt’s mechanisms for reconstructing cell position and size (currently it’s using cell index, domain edges, refinement level + a `refine_by` multiplier) so that cell edges can be stored as arrays that frontends would need to somehow retrieve from datafile or reconstruct from parameter files.
>>>>>
>>>>> There’s been some effort in the past to get over this limitation led by John and Matt, and we’ve been hitting this wall from time to time in discussions.
>>>>> Now, I understand that this would most likely not be a trivial change, especially because we don’t want to break existing frontends in the process (or performance !), and at the moment, I don’t have a very clear idea of what exact steps are needed to achieve this goal, and in particular I don’t have enough material to draft a YTEP on the topic.
>>>>> It seems clear we’d need *some* refactoring/generalisation of Index classes and friend classes and new pixelizer routines would likely be needed. Anything else ?
>>>>>
>>>>> I however have personal motivation to move forward here since my team is developing an hydro code and running simulations where grid streching is the rule rather than the exception.
>>>>>
>>>>> What I would like to do to get us rolling is simply collect the existing issues and PRs related to this goal. I know there are a couple, but it’s very easy to loose track of them, and it’s hard to see the big picture.
>>>>> Can I create a label (“arbitrary grids” ?) and apply it to existing material touching this topic ?
>>>>> Do you know of existing PRs and issues that are relevant to this topic ?
>>>>>
>>>>> Here are the ones I know:
>>>>> - https://github.com/yt-project/yt/issues/3481
>>>>> - https://github.com/yt-project/yt/pull/2998
>>>>> - https://github.com/yt-project/yt/issues/1880
>>>>> - https://github.com/yt-project/yt/issues/385 (from 2012 !!)
>>>>>
>>>>> Thanks
>>>>> Clément
>>>>> _______________________________________________
>>>>> yt-dev mailing list -- yt-dev(a)python.org
>>>>> To unsubscribe send an email to yt-dev-leave(a)python.org
>>>>> https://mail.python.org/mailman3/lists/yt-dev.python.org/
>>>>> Member address: michael.zingale(a)stonybrook.edu
>>>
>>> _______________________________________________
>>> yt-dev mailing list -- yt-dev(a)python.org
>>> To unsubscribe send an email to yt-dev-leave(a)python.org
>>> https://mail.python.org/mailman3/lists/yt-dev.python.org/
>>> Member address: michael.zingale(a)stonybrook.edu
>>
>> --
>>
>> Michael Zingale
>> Professor of Physics and Astronomy
>>
>> Dept. of Physics & Astronomy • Stony Brook University • Stony Brook, NY 11794-3800
>> phone: 631-632-8225
>> e-mail: michael.zingale(a)stonybrook.edu
>> web: [https://zingale.github.io](https://zingale.github.io/)
>> github: [https://github.com/zingale](http://github.com/zingale)
Dear yt dev team,
As was previously mentioned in yt 4.0.0's release notes, we're currently
planning to drop support for Python 3.6 in our next feature release (4.1.0)
I propose that this change be performed on the dev branch of yt by late
October this year, but not earlier than October the 20th.
As was discussed on Slack, I would like to send an official announcement
to yt-users(a)python.org, but I’ll leave open to discussion for a week first, or
as long as necessary for the discussion to resolve happily.
—
Here's the PR to drop Python 3.6 https://github.com/yt-project/yt/pull/2917
Please take minute to either request changes, discuss and/or approve it.
—
Nice implications
This means we'll soon be able to use Python 3.7+ only syntax and features.
Here are the two key highlights that I personally think are most impactful for yt:
- `from __future__ import annotations` can be used to leverage modern Python annotations features from more recent versions (as late as 3.11 !) This will be extremely useful to improve typing in yt, and in support to YTEP 0038 https://github.com/yt-project/ytep/pull/17 Here's a gist with some showcase examples of how this __future__ import helps simplifying typed code https://gist.github.com/neutrinoceros/0eaa046a00689254dee0738c65c9bdfc
- the `dataclasses` module becomes available https://docs.python.org/3/whatsnew/3.7.html#whatsnew37-pep557
See the detailed "what's new" note from Python 3.7.0 for more https://docs.python.org/3/whatsnew/3.7.html
Additional notes
- Python 3.6 is 5 years old and will be EOL (end of life) by December https://www.python.org/dev/peps/pep-0494/
- Numpy dropped support for Python 3.6 in numpy 1.20.0, released in January 2021 https://github.com/numpy/numpy/releases/tag/v1.20.0
- Matplotlib dropped support fot Python 3.6 in matplotlib 3.4.0, released in March 2021 https://github.com/matplotlib/matplotlib/commit/f9907b49f9d00c03cf014a25d05…
- Python 3.10 will be officially released early October https://www.python.org/dev/peps/pep-0619/
- Adding support for 3.10 should be a breeze^{TM}: we've been running a subset of our test suite against it (+ future versions of matplotlib and numpy) for a couple months, so the expected most important bottleneck for us will depend on when numpy and matplotlib are able to publish wheels for 3.10
- The plan is to first add support to 3.10 and *then* drop 3.6.
- This is in agreement with NEP 29, which was coauthored by our own release manager, Madicken Munk https://numpy.org/neps/nep-0029-deprecation_policy.html
- Various 3.7+ idioms will now be continuously enforced with pyupgrade https://github.com/asottile/pyupgrade which is already part of our pre-commit toolbelt.
Cheers, and have a great week
Clément
Hi folks
As you may be aware, yt’s grid based index classes offer support for patched-based AMR as well octrees, but it’s currently lacking support for so called “stretched gridding scheme”. This is a known limitation in several existing frontends (AMRVAC, BoxLib, Chombo-Pluto are the ones I know of).
For context, what I refer to as “stretched grids” are grids where cell sizes are not simple powers of 1/2 (with respect to a unitary reference) as in classic AMR and octrees, but may be specified by arbitrary rules (such as geometric series). Some hydro code support having parts of the grid following classic AMR scheme and other blocks to be “stretched” along one or several directions at once. This is the case of AMRVAC and Pluto, for instance. BoxLib also has support for complex refinement rules that are not covered by storing a single `refine_by` integer as we’re currently doing in Index classes (this is https://github.com/yt-project/yt/issues/3481).
It seems to me that the “easy” way of supporting these data formats would be to generalise yt’s mechanisms for reconstructing cell position and size (currently it’s using cell index, domain edges, refinement level + a `refine_by` multiplier) so that cell edges can be stored as arrays that frontends would need to somehow retrieve from datafile or reconstruct from parameter files.
There’s been some effort in the past to get over this limitation led by John and Matt, and we’ve been hitting this wall from time to time in discussions.
Now, I understand that this would most likely not be a trivial change, especially because we don’t want to break existing frontends in the process (or performance !), and at the moment, I don’t have a very clear idea of what exact steps are needed to achieve this goal, and in particular I don’t have enough material to draft a YTEP on the topic.
It seems clear we’d need *some* refactoring/generalisation of Index classes and friend classes and new pixelizer routines would likely be needed. Anything else ?
I however have personal motivation to move forward here since my team is developing an hydro code and running simulations where grid streching is the rule rather than the exception.
What I would like to do to get us rolling is simply collect the existing issues and PRs related to this goal. I know there are a couple, but it’s very easy to loose track of them, and it’s hard to see the big picture.
Can I create a label (“arbitrary grids” ?) and apply it to existing material touching this topic ?
Do you know of existing PRs and issues that are relevant to this topic ?
Here are the ones I know:
- https://github.com/yt-project/yt/issues/3481
- https://github.com/yt-project/yt/pull/2998
- https://github.com/yt-project/yt/issues/1880
- https://github.com/yt-project/yt/issues/385 (from 2012 !!)
Thanks
Clément
Hi folks,
I made an issue about using a backport service for backporting
bugfixes. I would like to request comments on it go to that issue,
and I will summarize here if we reach a conclusion.
https://github.com/yt-project/yt/issues/3553
-Matt
Hi yt-dev !
I hope you’re all doing well.
I would like to propose we plan for another bug fix release for the yt 4.0.x series. We had approximately 20 merged bug fixes since yt 4.0.1 (excluding docs and infrastructure fixes), and a couple bug fixing PRs that could realistically go in soon.
So how about we start a 4.0.2 milestone to keep track of our progress with backports ?
This time around, they’ll have to be manually performed, which I believe is one reason why we don’t do this more often, but in the future, we could automate the back porting process. For instance, Matplotlib and Ipython are using this tool: https://meeseeksbox.github.io
As far as I understand, this bot can create backport PRs mirroring patches associated with a milestone, itself attached to dedicated release branch.
Let me know if you have any thoughts on that too.
Cheers,
Clément
Hi folks,
Clement has prepared a rather large and expansive pull request that
adds type hinting to the current yt codebase. I have read and
reviewed it, and I find it to be an improvement over our current
situation, *without* causing any backward-incompatible issues.
https://github.com/yt-project/yt/pull/3546
It adds a few things that I'd like to highlight:
1. mypy checking, which someone else can explain better than I can,
but which checks that we are in fact passing the types to different
functions that we think we are.
2. better support for hinting at variable types in IDEs such a VS
Code, to reduce the likelihood of errors
3. a system for updating our usage of type hints in a forward compatible way
I would like to propose that once it has been marked as non-draft, we
prepare ourselves to merge it. While adding type hinting to a
codebase can be subject to a number of debates -- which I have every
intention of participating in! -- I would like to propose that we try
to prepare ourselves to lower the barrier for this particular set of
changes, to avoid being put in a situation where everyone agrees on
the end result but the latency caused by decision making around
specifics causes the decision to languish and continue indefinitely.
So I'd ask that folks that are interested in the *first* (but
absolutely *not final*) pass at adding type hints examine the pull
request and provide feedback, and we move forward with it
expeditiously.
(For instance, as folks have seen in the "particle reader cleanup"
pull request, I have a difference of opinion with Clement about some
specifics -- but I do not want that to get in the way of the first
iteration being sealed off and brought into the code base.)
-Matt
Hi all,
I've been tidying up the PRs a bit. I wanted to share a couple things:
1. I've added branch protections. This means that, except for
administrators (we can also make administrators do it) it now mandates
that these status checks be satisfied before merging:
- "Mergeable"
- "WIP"
- "py38"
(This is not the full set -- it notably excludes OS specific builds
and whatnot. But this will catch almost all the main things we reject
PRs for. It would also be possible to add *everything* to this list,
but I wasn't sure if that was warranted.)
2. All pull requests have to come from a branch. We mostly mandate
this already.
3. All PRs require at least one approval. (There are more
fine-grained protections we could add if we were to develop on a
branch other than main.)
This now lets us use what's called "auto-merge," This means that on
any PR that does not have these *yet* met (specifically, status
checks) we can say, "if this passes, merge it." This should improve
the efficiency of large PR triages, as it means that we can review,
queue things for merging and then it will merge *if-and-only-if* those
tests pass.
I'm going to propose a couple other minor-ish changes to help
streamline the workflow, especially for changelogs, but the main ones
there are that I'm going to propose (once I've written it up) changing
the labels a little bit and also freely editing issue titles.
-Matt
Dear yt users,
As was previously mentionned in yt 4.0.0's release notes, we're planning to drop
support for Python 3.6 in our next feature release (4.1.0). The change is set to
happen on the dev branch of yt not earlier than October the 20th, this year.
This means that if you wish to use our most recent features and bugfixes past this date, you'll
need to make sure you're running Python 3.7 or newer. Note that the
4.0.x series will retain compatibility with Python 3.6.
The exact method to upgrade your Python runtime depends on what tool(s) you used to
install Python or to manage your environments. In general, please lookup the
method that applies to your tools. Here is the documentation for some common
methods:
- using venv (see "--upgrade" flag) https://docs.python.org/3/library/venv.html
- using conda https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-pyth…
In general, yt is primarily tested with the latest Python versions available, so
we recommend upgrading Python to the latest version available to you.
There will be a reminder in two weeks from now.
Note that we're also planning to officially support Python 3.10 as soon as possible,
but likely not earlier than mid-October.
Have a great day,
Clément Robert, on behalf of the yt dev team
Hi yt-dev,
When I run a simple yt code, I run into the error message"ValueError: byte must be in range(0, 256) "
It was happen at "yt\data_objects\static_output.py", line 249, in unique_identifier,name_as_bytes =bytearray(map(ord, self.parameter_filename)), and then I went to the source code to see what happen. Although I tried to find the problem, I had no idea why the array would become so large.
I used ramses to produce the outputfile,info_00001.txt.
Here is my code:
import yt
import numpy as np
import matplotlib.pyplot as plt
# loading the snapshot
ds = yt.load("info_00001.txt") ##it is ramses output
ad = ds.all_data()
# reading particle positions and velocities into numpy arrays
pos = ad['particle_position']
vel = ad['particle_velocity']
# visualizing particle positions (ugly way)
plt.scatter(pos[:,0], pos[:,1], s=0.001)
plt.savefig('simple_visualization.png')
# writing particle positions to a csv file, which can be read
# into datashader
f = open('pos.csv', 'w')
f.write('x_col,y_col,z_col\n')
for i in pos:
f.write('%f,%f,%f\n' % (i[0], i[1], i[2]))
