tl;dr: We are working on improving support and performance for particle data in yt. We want feedback on our ideas and ask for people uninvolved in this effort to do code reviews and participate in design discussions and workflow discussions going forward

Last week we had a sprint here at NCSA around the "demeshening" idea a few of us have been throwing around. The basic idea here is to incorporate the code Meagan Lang has been working on over the past year into yt, allowing improved performance for particle codes. One consequence of this work is that there will no longer be a space-filling octree mesh for SPH data, instead the data will be indexed by local mesh patches built on compressed EWAH bitmap indices

As part of this sprint, we had lots of discussions about how this work might proceed, and people also did some work on moving several aspects of this effort forward:

I started work on an SPH pixelizer we can use for SlicePlot and ProjectionPlot:

Bili Dong started working on a ray data object that can be used to generate absorption spectra from SPH data without depositing data onto an octree mesh:

And Matt Turk and Alex Lindsay worked on improvements for the field system:

Alex's pull request also received some love today and I'd like to request review on it, as it will unblock further work on the pixelizers. In particular, I'd like to build on top of Alex's pull request to add a new kind of sampling_type, "local" which correspond to fully local fields. This generalizes the current default situation (i.e. the field does not have a ValidateSpatial validator). My ultimate goal here is to make it easier to support SPH particle fields. For fields that are fully local, we don't need to create different implementations for the 'cell' and 'particle' samping_type, instead we can just use the same implementation for all field types.

Once that is in, I think I'll be able to make it so projections and slices work with the majority of yt derived fields using the new SPH pixelizers I've added in PR 2382:

Unfortunately PR 2382 won't be mergeable at that point because we also need to come up with a solution for fields that require finite differences (this is discussed a bit in one of Daniel Price's papers, starting around section 3.3.2: http://arxiv.org/pdf/1012.1885v1.pdf)

I think once we've agreed on the way SPH particle fields should work going forward (I'm currently drafting a YTEP for this) then the demeshening work can build on top of that in a piecemeal fashion.

If we can I'd like it if we try to upstream stuff early and often to make this work easier to review and to improve testing by users and other developers. I don't want to move development into another fork of yt or a new named branch, since we had issues with that in the past. This approach might add some work we wouldn't have otherwise done (i.e. we might need to add compatibility layers that will eventually be torn down), but it will help keep us honest and ensure we're getting reviews from people uninvolved in this effort.

That said, if you'd like to try out the new code or even chip in, I'm happy to pull changes from people's forks or to review pull requests against my fork of yt.

I think that's it, sorry for the novel! I'd very much appreciate comments or questions about this.

-Nathan