Mailman 3 May 2018 - yt-dev

Creating a yt-4.0 branch in the main repo
by Nathan Goldbaum 23 May '18

23 May '18

Hi all, Currently there is active ongoing work in my fork of yt on the "sph-viz" branch. This branch contains the bitmap index work and accompanying changes to the particle selection API. Collectively these changes constitute a major backward incompatible change and thus cannot be easily integrated into the master branch on the main repo without major disruption. Doing that would also prevent us from being able to do a 3.5 or 3.6 release (currently there's no plan for a 3.6 release but crazier things have happened). We've already come to some level of consensus both here and in YTEP discussions that we want to make some major backward incompatible changes to yt in yt-4.0 and that the demeshening work should be one of those changes. To ease development going forward and make it clearer both how to get started and how to contribute, I'd like to push the current state of the "sph-viz" branch to the main yt repo and rename the branch "yt-4.0". Afterwards there will be "yt-2.x", "master", "stable", and "yt-4.0" branches in the main repo. Most pull requests will still go to master, but pull requests that implement changes or fixes that are specific to the demeshening or for planned features in yt-4.0 will go to the yt-4.0 branch. I'd additionally like to suggest that the threshold for merging code to the yt-4.0 branch should be lower than in master, possibly by reducing the number of required reviewers to one. I'd prefer not to make it harder to merge code than that to keep development velocity high and to make sure we're not blocked on getting code merged in for lack of reviewers who are familiar with the changes in the yt-4.0 branch. I don't think it's necessary to create a new repository, since the master branch will remain the default branch. People will only get the experimental yt-4.0 branch if they explicitly fetch it and check it out. As for testing, I think we'd just need to set up something in Jenkins to check the yt-4.0 branch as well as master for new PRs. The cloud CI services should "just work". We may end up turning off answer testing if it's too hard to have two separate sets of test answers for master and yt-4.0. Eventually, just like for yt 3.0 if you remember that, we will merge the yt 4.0 branch in. Doing that will require ensuring the documentation has been updated and developing community consensus that the new features are ready for widespread testing among people running on the yt master branch. Looking forward to hearing what people think about this, -Nathan Goldbaum

6 9

soliciting thoughts on implementing evolving frontend
by Britton Smith 22 May '18

22 May '18

Hi everyone, I'm working on adding/updating yt's support for Enzo-P data. If you're unfamiliar, Enzo-P is a totally different data format than Enzo. Enzo-P is still under heavy development, but is approaching the point where it can be used for science. I added an initial frontend last year to support grid data and this open PR <https://github.com/yt-project/yt/pull/1490> adds support for particles and cosmological data sets. This new PR also accommodates changes to the field naming conventions and the ability to get simulation parameters from a libconfig formatted file. Support for these changes is currently implemented a way that respects backward compatibility, which is mostly for the initial sample datasets that accompanied the addition of this frontend. It is likely that the file format will continue to evolve over the next months as we converge on the best design. Since this code is still mostly in the development stage, I would prefer to not have to maintain backward compatibility with prior stages of this evolution as there is almost assuredly no data existing in those forms other than the sample data in yt-project.org/data. Trying to do that will only introduce more if statements and confusion in the code. What I am asking is if I can be allowed to hit the reset button for this frontend and support only the data being produced by the current tip of Enzo-P. This would mean replacing existing sample data, but that would allow me to already make considerable simplifications to the frontend. In addition, I would like to be able to do this once or twice again in the near future as things get finalized, but before the code starts seeing wider public use. I realize this is a break from our pledge to be backward compatible as much as possible, which is why I'm bringing it up here. If periodically altering the frontend in this way is off limits, then an alternative could be moving Enzo-P support into an external yt extension package until the format is converged. I'm happy to do that, too. I would prefer if the open PR not sit and sit, either being accepted sooner than later with the understanding that things may change or being declined so I can put that functionality elsewhere. Any thoughts would be appreciated. Britton

4 4

GSoC Blog
by Ashley Kelly 18 May '18

18 May '18

Hi everyone, I'm just introducing myself as one of the successful Google Summer of Code students that will be working with yt this summer. I will be working with Nathan and Matt on aspects of the demeshening project. This will have a particular focus on developing an interface to calculate the value of an interpolated particle field at an arbitrary location using a couple of approaches. You can learn more about my project and follow my progress here. <https://ashkelly.github.io/gsoc-blog/> I will attempt to update the blog at least every couple of weeks. Feel free to ask any questions! Ash

1 0

Making Slack more useful
by Nathan Goldbaum 18 May '18

18 May '18

Hi all, I just had a discussion with Molly Peeples here at the Enzo workshop. She pointed out to me that the yt slack is currently very noisy and active and that there are a number of ways we could try to make Slack a more useful resource for both the user and developer community. This is informed by her experience in several very busy slack organizations. I've summarized her suggestions below. 1. Make the general channel less noisy Currently there are lots of discussions that go on in general. This is ok for a smaller slack, but for a slack community of our size this just encourages people to silence all activity from the yt slack and then never participate again. Making #general quieter will encourage people to check the yt slack when there's a major update and then perhaps more infrequently check in on other channels. Another thing we can do immediately to make #general less noisy is make our github integration publish new pull requests and issues to a new #development or #github channel. 2. Point people asking for help to the #help channel. Since people join #general automatically when they first join, there will inevitably be questions getting asked in #general. We should point people, both in the #general channel topic, and actively via social interactions on Slack to the #help channel. 3. Use threads more often This way an ongoing conversation does not generate tons of notifications. This also means that if someone responds to your question in a thread, you should use that thread to reply. 4. Create new channels often If there seems to be a topic that is generating lots of discussion, we should feel less inhibited to mention that this is happening and try to move discussion about that topic to a new channel. This way people who are interested in that topic get notifications about that topic and everyone else can ignore it. Ultimately my hope here is that more people will participate in our slack community and eventually in the yt community as a whole. Please don't take this e-mail as a dictate from me about how we should use slack but instead as the beginning of a discussion about this. I'd especially like to hear from people who currently don't use much slack because it's too noisy and from people who think my above suggestions are annoying or overbearing. Do you think you'd use Slack more if it were less noisy? Are there other ways we can improve the hygiene of our Slack community? Thanks for your time and attention, Nathan Goldbaum

7 14

GSoC with yt | https://git-abhishek.github.io/blog/
by Abhishek Singh 18 May '18

18 May '18

Hi everyone, I am thankful to all the yt community members for selecting me as one of the two students to work as a Google Summer of Code student. The program officially started on May 14, 2018 and will end by Aug 6, 2018. During this period I will be working on the project "Improve test coverage and test performance". Here is the *blog post url <https://git-abhishek.github.io/blog/>*showing my progress. I aim to write regularly (bi-weekly) showing my learning/issues/progress. Let me know, where I can find yours! I welcome all the suggestions or any other comments that could help us make yt better. My brief background: I just completed my first year out of two of my Masters in Computer Science from University of Massachusetts, Amherst. I did my undergraduate from BITS Pilani (India) in Computer Science and Masters in Mathematics. I have two and a half years of software development experience at Citi (full-time), Software Robotics Corp (full-time) and Amazon(internship). Looking forward to an exciting summer! Best, Abhishek Singh

1 0

Reproduce a memory leak
by Nathan Goldbaum 07 May '18

07 May '18

Hi all, If anyone has some time to check, I'd appreciate it if you could let me know if you can reproduce the memory leak described by Rixin Li in this issue: https://github.com/yt-project/yt/issues/1776 To check, just do the following: $ curl -JO http://use.yt/upload/33b7e323 $ tar xzvf yt_load_memory_leak_issue1776.tar.gz $ cd yt_load_memory_leak_issue1776 $ python encapsulated_yt.py (the tarball is only half a megabyte) On my machine that script has a peak memory usage of ~180 MB or so and the garbage collector is able to destroy the dataset objects that get created. However, Rixin reports linearly increasing memory usage with time, which seems to be because the dataset objects or something created by the datasets can't be garbage collected. Unfortunately it's difficult to remotely debug issues like this so it would be very nice to find someone else who can trigger the memory leak. Thanks for your help, Nathan

4 3