Currently there is active ongoing work in my fork of yt on the "sph-viz"
branch. This branch contains the bitmap index work and accompanying changes
to the particle selection API. Collectively these changes constitute a
major backward incompatible change and thus cannot be easily integrated
into the master branch on the main repo without major disruption. Doing
that would also prevent us from being able to do a 3.5 or 3.6 release
(currently there's no plan for a 3.6 release but crazier things have
We've already come to some level of consensus both here and in YTEP
discussions that we want to make some major backward incompatible changes
to yt in yt-4.0 and that the demeshening work should be one of those
To ease development going forward and make it clearer both how to get
started and how to contribute, I'd like to push the current state of the
"sph-viz" branch to the main yt repo and rename the branch "yt-4.0".
Afterwards there will be "yt-2.x", "master", "stable", and "yt-4.0"
branches in the main repo. Most pull requests will still go to master, but
pull requests that implement changes or fixes that are specific to the
demeshening or for planned features in yt-4.0 will go to the yt-4.0 branch.
I'd additionally like to suggest that the threshold for merging code to the
yt-4.0 branch should be lower than in master, possibly by reducing the
number of required reviewers to one. I'd prefer not to make it harder to
merge code than that to keep development velocity high and to make sure
we're not blocked on getting code merged in for lack of reviewers who are
familiar with the changes in the yt-4.0 branch.
I don't think it's necessary to create a new repository, since the master
branch will remain the default branch. People will only get the
experimental yt-4.0 branch if they explicitly fetch it and check it out.
As for testing, I think we'd just need to set up something in Jenkins to
check the yt-4.0 branch as well as master for new PRs. The cloud CI
services should "just work". We may end up turning off answer testing if
it's too hard to have two separate sets of test answers for master and
Eventually, just like for yt 3.0 if you remember that, we will merge the yt
4.0 branch in. Doing that will require ensuring the documentation has been
updated and developing community consensus that the new features are ready
for widespread testing among people running on the yt master branch.
Looking forward to hearing what people think about this,
I'm working on adding/updating yt's support for Enzo-P data. If you're
unfamiliar, Enzo-P is a totally different data format than Enzo. Enzo-P is
still under heavy development, but is approaching the point where it can be
used for science. I added an initial frontend last year to support grid
data and this open PR <https://github.com/yt-project/yt/pull/1490> adds
support for particles and cosmological data sets. This new PR also
accommodates changes to the field naming conventions and the ability to get
simulation parameters from a libconfig formatted file. Support for these
changes is currently implemented a way that respects backward
compatibility, which is mostly for the initial sample datasets that
accompanied the addition of this frontend.
It is likely that the file format will continue to evolve over the next
months as we converge on the best design. Since this code is still mostly
in the development stage, I would prefer to not have to maintain backward
compatibility with prior stages of this evolution as there is almost
assuredly no data existing in those forms other than the sample data in
yt-project.org/data. Trying to do that will only introduce more if
statements and confusion in the code.
What I am asking is if I can be allowed to hit the reset button for this
frontend and support only the data being produced by the current tip of
Enzo-P. This would mean replacing existing sample data, but that would
allow me to already make considerable simplifications to the frontend. In
addition, I would like to be able to do this once or twice again in the
near future as things get finalized, but before the code starts seeing
wider public use.
I realize this is a break from our pledge to be backward compatible as much
as possible, which is why I'm bringing it up here. If periodically altering
the frontend in this way is off limits, then an alternative could be moving
Enzo-P support into an external yt extension package until the format is
converged. I'm happy to do that, too. I would prefer if the open PR not sit
and sit, either being accepted sooner than later with the understanding
that things may change or being declined so I can put that functionality
Any thoughts would be appreciated.
I'm just introducing myself as one of the successful Google Summer of
Code students that will be working with yt this summer.
I will be working with Nathan and Matt on aspects of the demeshening
project. This will have a particular focus on developing an interface to
calculate the value of an interpolated particle field at an arbitrary
location using a couple of approaches.
You can learn more about my project and follow my progress here.
I will attempt to update the blog at least every couple of weeks.
Feel free to ask any questions!
I just had a discussion with Molly Peeples here at the Enzo workshop. She
pointed out to me that the yt slack is currently very noisy and active and
that there are a number of ways we could try to make Slack a more useful
resource for both the user and developer community. This is informed by her
experience in several very busy slack organizations. I've summarized her
1. Make the general channel less noisy
Currently there are lots of discussions that go on in general. This is ok
for a smaller slack, but for a slack community of our size this just
encourages people to silence all activity from the yt slack and then never
participate again. Making #general quieter will encourage people to check
the yt slack when there's a major update and then perhaps more infrequently
check in on other channels.
Another thing we can do immediately to make #general less noisy is make our
github integration publish new pull requests and issues to a new
#development or #github channel.
2. Point people asking for help to the #help channel.
Since people join #general automatically when they first join, there will
inevitably be questions getting asked in #general. We should point people,
both in the #general channel topic, and actively via social interactions on
Slack to the #help channel.
3. Use threads more often
This way an ongoing conversation does not generate tons of notifications.
This also means that if someone responds to your question in a thread, you
should use that thread to reply.
4. Create new channels often
If there seems to be a topic that is generating lots of discussion, we
should feel less inhibited to mention that this is happening and try to
move discussion about that topic to a new channel. This way people who are
interested in that topic get notifications about that topic and everyone
else can ignore it.
Ultimately my hope here is that more people will participate in our slack
community and eventually in the yt community as a whole.
Please don't take this e-mail as a dictate from me about how we should use
slack but instead as the beginning of a discussion about this. I'd
especially like to hear from people who currently don't use much slack
because it's too noisy and from people who think my above suggestions are
annoying or overbearing. Do you think you'd use Slack more if it were less
noisy? Are there other ways we can improve the hygiene of our Slack
Thanks for your time and attention,
I am thankful to all the yt community members for selecting me as one of
the two students to work as a Google Summer of Code student.
The program officially started on May 14, 2018 and will end by Aug 6, 2018.
During this period I will be working on the project "Improve test coverage
and test performance".
Here is the *blog post url <https://git-abhishek.github.io/blog/>*showing
my progress. I aim to write regularly (bi-weekly) showing my
learning/issues/progress. Let me know, where I can find yours! I welcome
all the suggestions or any other comments that could help us make yt better.
My brief background:
I just completed my first year out of two of my Masters in Computer Science
from University of Massachusetts, Amherst. I did my undergraduate from BITS
Pilani (India) in Computer Science and Masters in Mathematics. I have two
and a half years of software development experience at Citi (full-time),
Software Robotics Corp (full-time) and Amazon(internship).
Looking forward to an exciting summer!
If anyone has some time to check, I'd appreciate it if you could let me
know if you can reproduce the memory leak described by Rixin Li in this
To check, just do the following:
$ curl -JO http://use.yt/upload/33b7e323
$ tar xzvf yt_load_memory_leak_issue1776.tar.gz
$ cd yt_load_memory_leak_issue1776
$ python encapsulated_yt.py
(the tarball is only half a megabyte)
On my machine that script has a peak memory usage of ~180 MB or so and the
garbage collector is able to destroy the dataset objects that get created.
However, Rixin reports linearly increasing memory usage with time, which
seems to be because the dataset objects or something created by the
datasets can't be garbage collected.
Unfortunately it's difficult to remotely debug issues like this so it would
be very nice to find someone else who can trigger the memory leak.
Thanks for your help,