New issue 1336: Field parameters that yt doesn't know about are assumed to be dimensionless
Consider the following script:
from yt.fields.derived_field import ValidateParameter
def _overdensity(field, data):
rhobar = data.get_field_parameter('rhobar')
return data[('ramses', 'Density')]/rhobar - 1.
ds = yt.load('output_00080/info_00080.txt')
ad = ds.all_data()
rhobar = ad.mean(('ramses', 'Density'))
ds.add_field(("gas", "overdensity"), function=_overdensity,
Right now this dies with the following traceback:
Traceback (most recent call last):
File "test.py", line 16, in <module>
File "/usr/local/lib/python3.5/site-packages/yt/data_objects/static_output.py", line 1090, in add_field
deps, _ = self.field_info.check_derived_fields([name])
File "/usr/local/lib/python3.5/site-packages/yt/fields/field_info_container.py", line 362, in check_derived_fields
fd = fi.get_dependencies(ds = self.ds)
File "/usr/local/lib/python3.5/site-packages/yt/fields/derived_field.py", line 178, in get_dependencies
File "/usr/local/lib/python3.5/site-packages/yt/fields/field_detector.py", line 99, in __missing__
vv = finfo(self)
File "/usr/local/lib/python3.5/site-packages/yt/fields/derived_field.py", line 204, in __call__
dd = self._function(self, data)
File "test.py", line 6, in _overdensity
return data[('ramses', 'Density')]/rhobar - 1.
File "/usr/local/lib/python3.5/site-packages/yt/units/yt_array.py", line 927, in __sub__
ro = sanitize_units_add(self, right_object, "subtraction")
File "/usr/local/lib/python3.5/site-packages/yt/units/yt_array.py", line 163, in sanitize_units_add
raise YTUnitOperationError(op_string, inp.units, dimensionless)
yt.utilities.exceptions.YTUnitOperationError: The subtraction operator for YTArrays with units (code_density) and (1) is not well defined.
What's going wrong here is the field detection system doesn't know that `rhobar` is a possible name for a field parameter, so it assumes it must be dimensionless. We should make the way the field detection system handles field parameters a bit smarter.
I'd like to nominate Alex Lindsay as a project member. He has been working
on improving support for unstructured mesh data in general and the Exodus
II frontend in particular.
Currently Alex has about 70 commits in the main repo and has worked on
several pull requests.
We need two more seconds to officially add him as a member. Once that
happens I will update the relevant places on bitbucket, the website, and
the YTEP repo.
If you'd like to learn more about membership in the yt project, take a look
at YTEP-1776: http://ytep.readthedocs.io/en/latest/YTEPs/YTEP-1776.html
New issue 1334: Yt misinterpretes gizmo binary files of type one
[Gizmo](https://bitbucket.org/phopkins/gizmo) binary files of type 1 (oldest type, no hdf5, no 4 character block delimiters) are not read/recognized correctly by yt.
Gizmo has introduced two additional integer-sized blocks (`CHILD_ID` and `GENERATION_ID`) for **all** particle types (gas, halo, disk, bulge, stars, boundary). So the order of blocks is:
3. Particle IDs
4. **Child IDs**
5. **Generation IDs**
7. Internal Energy
If the `LONGIDS` flag was set in Gizmo, the two new blocks have the size of a `long long` in C. Currently yt misinterpretes the `CHILD_ID` block as mass.
A way to identify this is to skip the block that might be `CHILD_ID` or `MASS`, read the size of the next block and depending on that decide whether we have a Gizmo or Gadget file:
- The block after `MASS` should be Internal energy therefore have `sizeof(float)` or `sizeof(double)`¹ times number of **gas** particles
- The block after `CHILD_ID` should be `GENERATION_ID`, which has `sizeof(int)` or `sizeof(long long)` times the **total** number of particles
Most people use HDF5 or at least snap format 2 with Gizmo, but there are a few legacy Gadget tools that require format 1. It should be recognized and read correctly in yt.
¹Depending on whether the Gizmo was compiled with the `OUTPUT_IN_DOUBLEPRECISION` flag.
The yt steering committee (Hilary Egan, Nathan Goldbaum, Cameron
Hummels, Kacper Kowalik, Sam Skillman, Britton Smith, Matthew Turk and
John ZuHone) proposes that we move the hosting infrastructure for yt
from BitBucket to GitHub, and as a side effect, migrate the primary
version control system from mercurial to git.
This decision wasn’t taken lightly, and it’s with more than a little
sadness that I’m writing this email. Beneath the “next steps” stuff,
I’ve detailed some of the rationale behind this, and outlined some of
our thinking. First, however, I’ll outline the next steps and what
this means for the (overlapping, non-disjoint) developer and user
* This email is to serve as a solicitation of comments; if you wish to
just say, “+1” or “me too” or anything along those lines, please feel
free to not send that email. If you have reservations, concerns to
raise, suggestions for the conversion process itself, or opinions
about the outline, please do contribute.
* The conversion process and plans for migrating code, issues, etc
will be detailed in a YTEP. What will almost certainly happen is that
for a period of time, we’ll have mirrors on both GH and BB, but at any
given time only one will be open for pull requests and code review.
(i.e., there’ll be a switch-flipping at some point to move, but we’ll
ease into it.) We anticipate this will be on the scale of months, not
hours or days.
* We decide on a timeline for when we turn off pull requests on the
old repository and turn them on on a new one.
* We’ll be figuring out how to move infrastructure (fido) to GitHub as
part of the YTEP.
What this means for you as someone who contributes, or runs off the mainline:
* At some point in the future, you will need to issue pull requests to
a repository on GitHub.
* Code reviews will happen on GitHub, and our process may change slightly.
* The yt repository on BitBucket will, for the foreseeable future, be
available, and we will mirror changes from GH for at least the
mid-term future, potentially much longer. All previous code reviews
will be backed up, but for reference will continue to be available on
* If you auto-update and want to push changes back up, you’ll be
grabbing from another place; this likely means re-cloning.
What this means for someone who runs yt:
* If you are using nightly or stable builds from pip, anaconda or
conda-forge, you should have no interruption whatsoever.
* If you’re running from the source repository, you may or may not
need to re-clone (with git), depending on the timescale of the mirror
continuing to run.
Big Things To Discuss:
* When we do flip the BB->GH switch is not clear; there are arguments
to be made for both “right away” and “in the summer.” There are some
big in-flight PRs that maybe should go in before we move.
* We’ll need to decide about things like squashing commits, rebasing,
etc. That can happen later.
I suppose that’s it. Thanks for reading. Unless we hear big
objections, we’ll draft a YTEP in the next week or two with some more
information, and then use that as an opportunity to finalize a
-Matt, on behalf of the yt steering committee
I’ll start this out with a personal comment. Discussions about this
have happened from time to time. I’d always, always pushed back. (I
wasn’t the only one, but I was usually pretty vocal.) This time, I
was the one who brought it up.
My reasoning, which is certainly not original, is that there are two
main reasons we should move. I want to emphasize neither of these is
related to hg; in fact, I’m incredibly sad about hg, and only a little
sad about leaving bitbucket. I’ll probably still use hg-git to manage
git via hg, and I’ll still happily talk about how amazing hg is.
Reason 1: It’s an inclusion issue. There is evidence that any
additional barriers to entry dramatically reduce participation in the
development of open source projects, and there is evidence that these
barriers disproportionately affect underrepresented and minoritized
groups. We have seen a decline in new contributions to yt.
Furthermore, the regular contributors to yt do not match the diversity
of the astronomical community (which, despite our efforts to be
interdisciplinary, is still our original home.) Asking people to jump
through additional hurdles helps neither of these problems.
Reason 2: There is an ecosystem around GitHub that we are not able to
participate in. Things like depsy.org, the other astronomy or
physical sciences projects, and the like. The advantage of a
monoculture (lowering barriers to entry) is also the disadvantage of a
monoculture in terms of an ecosystem. By moving to GitHub, we will
gain access to this ecosystem.
I could go on at length about these two reasons, but I’ll just leave
it there. For me, the first reason is the vastly more important one,
but both are valid.
This decision wasn’t taken lightly, and a number of issues were raised
-- not the least of which is reducing disruption to people that are
either clearly “users” or “developers” as well as the community of
dev-users. One issue that has been brought up is that every “disrupt
people’s workflow” token we spend has a real cost, and we should not
use them lightly. What we arrived at, which was a consensus if not a
consensus that left everyone entirely happy, was that this case is
worth one of those tokens.
-Matt, on behalf of himself
New issue 1332: Abiity to read a small subset of files in a multipart GADGET binary snapshot
In many situations it so happens that the data of the GADGET snapshot output is extremely large (mine is around 1 TB) and is divided into many parts in binary files. It becomes necessary to sometimes use only a few of these files instead of the entire dataset maybe for running some initial analyzing test codes. Currently yt doesn't support that. I would be great if such an ability is added to yt.
I'd like to bring PR 2526 to your attention. It introduces small, but
significant changes to the way how the API reference is built. Most
notably: it generates a one html page per *module* with classes and
methods as subsections, instead of a one page per method (as we do now).
An example deployment can be browsed here:
There are couple of pros to this approach:
* The build time is greatly reduced. It shaves a 0.5h from the total
build time on our server, also makes it feasible to build the API doc
locally (it takes 2m on my laptop vs over 1h before)
* The lower number of pages creates a better searchindex. As an example
try searching for an 'AxisAlignedSlicePlot' phrase on 'docs/dev and' on
A one con that Nathan identified is that methods of a parent class are
not explicitly listed for derived classes (it can be also observed on
the aforementioned AxisAlignedSlicePlot's page). Instead there's a link
to the parent class definition at the top of doc entry (Bases:
yt.visualization.plot_window.PWViewerMPL for AxisAlignedSlicePlot).
For someone heavily relying on a previous API doc structure, that's
going to be something to get used to.
I'd love to hear from y'all, either on the ml or directly on the pull