Mailman 3 March 2017 - yt-dev

Issue #1336: Field parameters that yt doesn't know about are assumed to be dimensionless (yt_analysis/yt)
by Nathan Goldbaum 20 Mar '17

20 Mar '17

New issue 1336: Field parameters that yt doesn't know about are assumed to be dimensionless https://bitbucket.org/yt_analysis/yt/issues/1336/field-parameters-that-yt-d… Nathan Goldbaum: Consider the following script: ``` import yt from yt.fields.derived_field import ValidateParameter def _overdensity(field, data): rhobar = data.get_field_parameter('rhobar') return data[('ramses', 'Density')]/rhobar - 1. ds = yt.load('output_00080/info_00080.txt') ad = ds.all_data() rhobar = ad.mean(('ramses', 'Density')) ad.set_field_parameter('rhobar', rhobar) ds.add_field(("gas", "overdensity"), function=_overdensity, take_log=False, force_override=True, validators=[ValidateParameter('rhobar')]) print(ad['overdensity']) ``` Right now this dies with the following traceback: ``` Traceback (most recent call last): File "test.py", line 16, in <module> validators=[ValidateParameter('rhobar')]) File "/usr/local/lib/python3.5/site-packages/yt/data_objects/static_output.py", line 1090, in add_field deps, _ = self.field_info.check_derived_fields([name]) File "/usr/local/lib/python3.5/site-packages/yt/fields/field_info_container.py", line 362, in check_derived_fields fd = fi.get_dependencies(ds = self.ds) File "/usr/local/lib/python3.5/site-packages/yt/fields/derived_field.py", line 178, in get_dependencies e[self.name] File "/usr/local/lib/python3.5/site-packages/yt/fields/field_detector.py", line 99, in __missing__ vv = finfo(self) File "/usr/local/lib/python3.5/site-packages/yt/fields/derived_field.py", line 204, in __call__ dd = self._function(self, data) File "test.py", line 6, in _overdensity return data[('ramses', 'Density')]/rhobar - 1. File "/usr/local/lib/python3.5/site-packages/yt/units/yt_array.py", line 927, in __sub__ ro = sanitize_units_add(self, right_object, "subtraction") File "/usr/local/lib/python3.5/site-packages/yt/units/yt_array.py", line 163, in sanitize_units_add raise YTUnitOperationError(op_string, inp.units, dimensionless) yt.utilities.exceptions.YTUnitOperationError: The subtraction operator for YTArrays with units (code_density) and (1) is not well defined. ``` What's going wrong here is the field detection system doesn't know that `rhobar` is a possible name for a field parameter, so it assumes it must be dimensionless. We should make the way the field detection system handles field parameters a bit smarter. Responsible: ngoldbaum

1 0

Nominate Alex Lindsay as a project member
by Nathan Goldbaum 14 Mar '17

14 Mar '17

Hi all, I'd like to nominate Alex Lindsay as a project member. He has been working on improving support for unstructured mesh data in general and the Exodus II frontend in particular. Currently Alex has about 70 commits in the main repo and has worked on several pull requests. We need two more seconds to officially add him as a member. Once that happens I will update the relevant places on bitbucket, the website, and the YTEP repo. If you'd like to learn more about membership in the yt project, take a look at YTEP-1776: http://ytep.readthedocs.io/en/latest/YTEPs/YTEP-1776.html -Nathan

6 5

Issue #1335: Add second order support for QUADS (yt_analysis/yt)
by Alexander Lindsay 14 Mar '17

14 Mar '17

New issue 1335: Add second order support for QUADS https://bitbucket.org/yt_analysis/yt/issues/1335/add-second-order-support-f… Alexander Lindsay: MOOSE users fairly commonly use second order LAGRANGE polynomials on quad elements; want to natively support visualization on such elements without reverting to first order.

1 0

Issue #1334: Yt misinterpretes gizmo binary files of type one (yt_analysis/yt)
by Christoph Bischko 10 Mar '17

10 Mar '17

New issue 1334: Yt misinterpretes gizmo binary files of type one https://bitbucket.org/yt_analysis/yt/issues/1334/yt-misinterpretes-gizmo-bi… Christoph Bischko: [Gizmo](https://bitbucket.org/phopkins/gizmo) binary files of type 1 (oldest type, no hdf5, no 4 character block delimiters) are not read/recognized correctly by yt. Gizmo has introduced two additional integer-sized blocks (`CHILD_ID` and `GENERATION_ID`) for **all** particle types (gas, halo, disk, bulge, stars, boundary). So the order of blocks is: 1. Positions 2. Velocities 3. Particle IDs 4. **Child IDs** 5. **Generation IDs** 6. Masses 7. Internal Energy ... If the `LONGIDS` flag was set in Gizmo, the two new blocks have the size of a `long long` in C. Currently yt misinterpretes the `CHILD_ID` block as mass. A way to identify this is to skip the block that might be `CHILD_ID` or `MASS`, read the size of the next block and depending on that decide whether we have a Gizmo or Gadget file: - The block after `MASS` should be Internal energy therefore have `sizeof(float)` or `sizeof(double)`¹ times number of **gas** particles - The block after `CHILD_ID` should be `GENERATION_ID`, which has `sizeof(int)` or `sizeof(long long)` times the **total** number of particles Most people use HDF5 or at least snap format 2 with Gizmo, but there are a few legacy Gadget tools that require format 1. It should be recognized and read correctly in yt. ¹Depending on whether the Gizmo was compiled with the `OUTPUT_IN_DOUBLEPRECISION` flag.

1 0

Issue #1333: Exception Ignored from yt.geometry.particle_deposit.SumParticleField.process (yt_analysis/yt)
by Desika Narayanan 09 Mar '17

09 Mar '17

New issue 1333: Exception Ignored from yt.geometry.particle_deposit.SumParticleField.process https://bitbucket.org/yt_analysis/yt/issues/1333/exception-ignored-from Desika Narayanan: When loading a sub-region of a gizmo snapshot, and trying to create an arbitrary grid, the code gives (numerous prints of) an error: ``` Exception IndexError: 'Out of bounds on buffer access (axis 2)' in 'yt.geometry.particle_deposit.SumParticleField.process' ignored``` An example code that reproduces this is here: http://paste.yt-project.org/show/7075/ with the 8.5 GB snapshot here: http://use.yt/upload/57463d74 Responsible: ngoldbaum

1 0

ANN: yt 3.3.5 released
by Nathan Goldbaum 08 Mar '17

08 Mar '17

We are proud to announce the release of yt 3.3.5! yt (http://yt-project.org) is an open-source, community-developed toolkit for analysis and visualization of volumetric data. Version 3.3.5 is a regularly scheduled bugfix release including fixes for a number of issues reported since the release of yt 3.3.4. We urge all users to update to the latest version. If your python installation is managed via conda, update following: $ conda update -c conda-forge yt If your python installation is managed via pip, update following: $ pip install -U yt Finally, if you manage your yt installation by running out of a clone of the yt mercurial repository, update following: $ yt update We would like to thank the following people for their contributions to this release: Nathan Goldbaum Kacper Kowalik Andrew Myers John ZuHone A summary of changes in this release, along with a link to the pull request implementing the changes follow below: Fix for off-axis projections of multiple fields producing data only for the first field (PR 2502 <https://bitbucket.org/yt_analysis/yt/pull-requests/2502>) Don't attempt to parse inputs or probins files for generic BoxLib data (PR 2517 <https://bitbucket.org/yt_analysis/yt/pull-requests/2517>) Fix C compilation warnings (PR 2519 <https://bitbucket.org/yt_analysis/yt/pull-requests/2519>) Fix an error in the halos plot callback that was unintentionally introduced on the stable branch while backporting another bugfix for yt 3.3.4 (PR 2520 <https://bitbucket.org/yt_analysis/yt/pull-requests/2520>) Ensure transparent volume renderings can be composited (PR 2521 <https://bitbucket.org/yt_analysis/yt/pull-requests/2521>) Allow user customization of the mean molecular weight via a field parameter in the Gadget temperature field (PR 2523 <https://bitbucket.org/yt_analysis/yt/pull-requests/2523>) Fix syntax error in code example in the documentation (PR 2524 <https://bitbucket.org/yt_analysis/yt/pull-requests/2524>)

1 0

Changes to yt source code hosting
by Matthew Turk 07 Mar '17

07 Mar '17

Hi everyone, The yt steering committee (Hilary Egan, Nathan Goldbaum, Cameron Hummels, Kacper Kowalik, Sam Skillman, Britton Smith, Matthew Turk and John ZuHone) proposes that we move the hosting infrastructure for yt from BitBucket to GitHub, and as a side effect, migrate the primary version control system from mercurial to git. This decision wasn’t taken lightly, and it’s with more than a little sadness that I’m writing this email. Beneath the “next steps” stuff, I’ve detailed some of the rationale behind this, and outlined some of our thinking. First, however, I’ll outline the next steps and what this means for the (overlapping, non-disjoint) developer and user communities. What’s Next: * This email is to serve as a solicitation of comments; if you wish to just say, “+1” or “me too” or anything along those lines, please feel free to not send that email. If you have reservations, concerns to raise, suggestions for the conversion process itself, or opinions about the outline, please do contribute. * The conversion process and plans for migrating code, issues, etc will be detailed in a YTEP. What will almost certainly happen is that for a period of time, we’ll have mirrors on both GH and BB, but at any given time only one will be open for pull requests and code review. (i.e., there’ll be a switch-flipping at some point to move, but we’ll ease into it.) We anticipate this will be on the scale of months, not hours or days. * We decide on a timeline for when we turn off pull requests on the old repository and turn them on on a new one. * We’ll be figuring out how to move infrastructure (fido) to GitHub as part of the YTEP. What this means for you as someone who contributes, or runs off the mainline: * At some point in the future, you will need to issue pull requests to a repository on GitHub. * Code reviews will happen on GitHub, and our process may change slightly. * The yt repository on BitBucket will, for the foreseeable future, be available, and we will mirror changes from GH for at least the mid-term future, potentially much longer. All previous code reviews will be backed up, but for reference will continue to be available on BitBucket. * If you auto-update and want to push changes back up, you’ll be grabbing from another place; this likely means re-cloning. What this means for someone who runs yt: * If you are using nightly or stable builds from pip, anaconda or conda-forge, you should have no interruption whatsoever. * If you’re running from the source repository, you may or may not need to re-clone (with git), depending on the timescale of the mirror continuing to run. Big Things To Discuss: * When we do flip the BB->GH switch is not clear; there are arguments to be made for both “right away” and “in the summer.” There are some big in-flight PRs that maybe should go in before we move. * We’ll need to decide about things like squashing commits, rebasing, etc. That can happen later. I suppose that’s it. Thanks for reading. Unless we hear big objections, we’ll draft a YTEP in the next week or two with some more information, and then use that as an opportunity to finalize a timeline. -Matt, on behalf of the yt steering committee Appendix: Rationale I’ll start this out with a personal comment. Discussions about this have happened from time to time. I’d always, always pushed back. (I wasn’t the only one, but I was usually pretty vocal.) This time, I was the one who brought it up. My reasoning, which is certainly not original, is that there are two main reasons we should move. I want to emphasize neither of these is related to hg; in fact, I’m incredibly sad about hg, and only a little sad about leaving bitbucket. I’ll probably still use hg-git to manage git via hg, and I’ll still happily talk about how amazing hg is. Reason 1: It’s an inclusion issue. There is evidence that any additional barriers to entry dramatically reduce participation in the development of open source projects, and there is evidence that these barriers disproportionately affect underrepresented and minoritized groups. We have seen a decline in new contributions to yt. Furthermore, the regular contributors to yt do not match the diversity of the astronomical community (which, despite our efforts to be interdisciplinary, is still our original home.) Asking people to jump through additional hurdles helps neither of these problems. Reason 2: There is an ecosystem around GitHub that we are not able to participate in. Things like depsy.org, the other astronomy or physical sciences projects, and the like. The advantage of a monoculture (lowering barriers to entry) is also the disadvantage of a monoculture in terms of an ecosystem. By moving to GitHub, we will gain access to this ecosystem. I could go on at length about these two reasons, but I’ll just leave it there. For me, the first reason is the vastly more important one, but both are valid. This decision wasn’t taken lightly, and a number of issues were raised -- not the least of which is reducing disruption to people that are either clearly “users” or “developers” as well as the community of dev-users. One issue that has been brought up is that every “disrupt people’s workflow” token we spend has a real cost, and we should not use them lightly. What we arrived at, which was a consensus if not a consensus that left everyone entirely happy, was that this case is worth one of those tokens. -Matt, on behalf of himself

1 0

Issue #1332: Abiity to read a small subset of files in a multipart GADGET binary snapshot (yt_analysis/yt)
by Anonymous 05 Mar '17

05 Mar '17

New issue 1332: Abiity to read a small subset of files in a multipart GADGET binary snapshot https://bitbucket.org/yt_analysis/yt/issues/1332/abiity-to-read-a-small-sub… Anonymous: In many situations it so happens that the data of the GADGET snapshot output is extremely large (mine is around 1 TB) and is divided into many parts in binary files. It becomes necessary to sometimes use only a few of these files instead of the entire dataset maybe for running some initial analyzing test codes. Currently yt doesn't support that. I would be great if such an ability is added to yt. Responsible: ngoldbaum

1 0

RFC: updating API docs
by Kacper Kowalik 04 Mar '17

04 Mar '17

Hi all! I'd like to bring PR 2526 to your attention. It introduces small, but significant changes to the way how the API reference is built. Most notably: it generates a one html page per *module* with classes and methods as subsections, instead of a one page per method (as we do now). An example deployment can be browsed here: http://yt-project.org/docs/mock There are couple of pros to this approach: * The build time is greatly reduced. It shaves a 0.5h from the total build time on our server, also makes it feasible to build the API doc locally (it takes 2m on my laptop vs over 1h before) * The lower number of pages creates a better searchindex. As an example try searching for an 'AxisAlignedSlicePlot' phrase on 'docs/dev and' on 'docs/mock'. A one con that Nathan identified is that methods of a parent class are not explicitly listed for derived classes (it can be also observed on the aforementioned AxisAlignedSlicePlot's page). Instead there's a link to the parent class definition at the top of doc entry (Bases: yt.visualization.plot_window.PWViewerMPL for AxisAlignedSlicePlot). For someone heavily relying on a previous API doc structure, that's going to be something to get used to. I'd love to hear from y'all, either on the ml or directly on the pull request. Cheers, Kacper

2 2

Issue #1331: Happy Birthday Sis (yt_analysis/yt)
by Anonymous 04 Mar '17

04 Mar '17

New issue 1331: Happy Birthday Sis https://bitbucket.org/yt_analysis/yt/issues/1331/happy-birthday-sis Anonymous:

1 0