Hi all, Now that 2.4 is out, I wanted to update you on where this roadmap (from early June, two months ago today!) is. It slipped a bit, but I think we're still broadly on track. My current plan is to focus on development of the 3.0 branch/fork, which is located here: http://bitbucket.org/yt_analysis/yt-3.0 It is nearly functional for all major functions from the patch-based perspective, but I am still working on projections and a handful of other operations -- flushing data back to grids (for clump finding), masking of cells (for extracted regions), and a few other medium-level features. Once it has reached feature completeness for patch-based, I think it will be ready to have many of you start testing it, kicking the tires, maybe even moving your development there. I'll update this group as I make progress, but I'd also be very appreciative if anyone wants to start testing it with me and working on it. I'm quite excited about some aspects of it, and I've already been pleased with simplifications in several areas. We should still probably leave open the possibility of a 2.5 release, but with a much shorter changelog than 2.4 had! :) I'll be around in IRC as well as on the mailing list if anyone has any concerns, suggestions, or wants to get involved in this. -Matt On Thu, Jun 7, 2012 at 4:50 PM, Matthew Turk wrote:

Hi all,

Sam and I are working on issuing a PR about the volume rendering refactor. This is the last major code change to go in place for 2.4, pending the quad_proj changes being accepted or rejected. I'm still thinking about this refine_by issue, and that might also make it in. After the volume_refactor, it's just testing and documentation left:

https://bitbucket.org/yt_analysis/yt/issues?status=new&status=open&milestone=2.4

After 2.4 is released, I am going to shift my new development exclusively to the yt-3.0 branch, which is currently hosted here:

http://bitbucket.org/yt_analysis/yt-3.0

This will include geometry refactoring and many remaining de-Enzoifications of the code base. 3.0 is nearing a usable state, but will not be ready for production for a while. However, I think that allowing the codebases to diverge too rapidly will lead to problems: a fracturing. So once the 3.0 codebase can muster feature parity with 2.x, I am going to ask that all development on the 2.x branch cease, and all pull requests for features and new functionality be applied only to the 3.0 branch. In advance of this, I would like to solicit help: we've identified a number of relatively straightforward changes to this code base that can and should be made. For instance, the changing of field names. I'd also like to consider moving to a more spread-out source code layout, like one class per file. Changes like these make it difficult to do cross-branch merges and transplants, which is why I would like to save them for after feature parity has been reached and development on 2.x has halted.

Previous threads:

http://lists.spacepope.org/pipermail/yt-dev-spacepope.org/2012-February/0018... http://lists.spacepope.org/pipermail/yt-dev-spacepope.org/2012-February/0018... http://lists.spacepope.org/pipermail/yt-dev-spacepope.org/2012-February/0018... http://lists.spacepope.org/pipermail/yt-dev-spacepope.org/2012-March/001941.... http://lists.spacepope.org/pipermail/yt-dev-spacepope.org/2012-March/001949....

If anyone has any strong objections to this strategy, please reply.

Estimated Timeline:

* June 26, 2.4 release * July 30, 3.0 reaches rough feature parity * Fall sometime: 3.0 merged into main 'yt' branch * Some time after that: 3.0 released

Criteria for Feature Parity:

* All patch-based AMR codes support data region selection * Parallelism is functional for all parallel-capable routines * All patch-based AMR codes support projections, slices, cutting planes * All patch-based AMR codes support volume rendering * All patch-based AMR codes support halo finding * Octree-based AMR codes (RAMSES and ART) support data region selection * Octree-based AMR codes (RAMSES and ART) support slices, projections and cutting planes

If any of you are interested in helping with this reengineering effort, I would be extremely interested in collaborating. My goal for this effort is to focus on making yt a future-compatible code, and changing it from an organically grown system into one that we design based on the years of experience we all have now had with it.

-Matt

Sam, On Tue, Aug 7, 2012 at 11:51 AM, Sam Skillman wrote:

My goals for 2.5:

Re-written, modularized, amr_kdtree. This will allow for kd-tree decomp of generalized collections of grids as well as future optimization, as well as parallelization over arbitrary N processors, not just power of 2! Alpha channel integration for the rendering, allowing for easier control of opaque and transparent features in the same image.

This sounds really awesome.

Neither of these necessarily belong in the 3.0 development IMO, but I could be convinced to work in that sandbox. Both of these are working in an alpha state right now.

I agree, these don't *need* to be in 3.0, so let's wait till that branch is feature complete before I start selling you on moving the dev there. :) ...but ... do you think the walk and the decomp could be separated, so we could feed it an octree? In the 3.0 branch I've got an oct geometry handler which could potentially be instrumented with the necessary additional info to allow fast traversal... -Matt

Sam

On Tue, Aug 7, 2012 at 9:46 AM, Matthew Turk wrote:

...

Hi all,

Now that 2.4 is out, I wanted to update you on where this roadmap (from early June, two months ago today!) is. It slipped a bit, but I think we're still broadly on track.

My current plan is to focus on development of the 3.0 branch/fork, which is located here:

http://bitbucket.org/yt_analysis/yt-3.0

It is nearly functional for all major functions from the patch-based perspective, but I am still working on projections and a handful of other operations -- flushing data back to grids (for clump finding), masking of cells (for extracted regions), and a few other medium-level features. Once it has reached feature completeness for patch-based, I think it will be ready to have many of you start testing it, kicking the tires, maybe even moving your development there. I'll update this group as I make progress, but I'd also be very appreciative if anyone wants to start testing it with me and working on it. I'm quite excited about some aspects of it, and I've already been pleased with simplifications in several areas.

We should still probably leave open the possibility of a 2.5 release, but with a much shorter changelog than 2.4 had! :)

I'll be around in IRC as well as on the mailing list if anyone has any concerns, suggestions, or wants to get involved in this.

-Matt

On Thu, Jun 7, 2012 at 4:50 PM, Matthew Turk wrote:

...

Hi all,

Sam and I are working on issuing a PR about the volume rendering refactor. This is the last major code change to go in place for 2.4, pending the quad_proj changes being accepted or rejected. I'm still thinking about this refine_by issue, and that might also make it in. After the volume_refactor, it's just testing and documentation left:

https://bitbucket.org/yt_analysis/yt/issues?status=new&status=open&milestone=2.4

After 2.4 is released, I am going to shift my new development exclusively to the yt-3.0 branch, which is currently hosted here:

http://bitbucket.org/yt_analysis/yt-3.0

This will include geometry refactoring and many remaining de-Enzoifications of the code base. 3.0 is nearing a usable state, but will not be ready for production for a while. However, I think that allowing the codebases to diverge too rapidly will lead to problems: a fracturing. So once the 3.0 codebase can muster feature parity with 2.x, I am going to ask that all development on the 2.x branch cease, and all pull requests for features and new functionality be applied only to the 3.0 branch. In advance of this, I would like to solicit help: we've identified a number of relatively straightforward changes to this code base that can and should be made. For instance, the changing of field names. I'd also like to consider moving to a more spread-out source code layout, like one class per file. Changes like these make it difficult to do cross-branch merges and transplants, which is why I would like to save them for after feature parity has been reached and development on 2.x has halted.

Previous threads:

http://lists.spacepope.org/pipermail/yt-dev-spacepope.org/2012-February/0018...

http://lists.spacepope.org/pipermail/yt-dev-spacepope.org/2012-February/0018...

http://lists.spacepope.org/pipermail/yt-dev-spacepope.org/2012-February/0018...

http://lists.spacepope.org/pipermail/yt-dev-spacepope.org/2012-March/001941....

http://lists.spacepope.org/pipermail/yt-dev-spacepope.org/2012-March/001949....

If anyone has any strong objections to this strategy, please reply.

Estimated Timeline:

* June 26, 2.4 release * July 30, 3.0 reaches rough feature parity * Fall sometime: 3.0 merged into main 'yt' branch * Some time after that: 3.0 released

Criteria for Feature Parity:

* All patch-based AMR codes support data region selection * Parallelism is functional for all parallel-capable routines * All patch-based AMR codes support projections, slices, cutting planes * All patch-based AMR codes support volume rendering * All patch-based AMR codes support halo finding * Octree-based AMR codes (RAMSES and ART) support data region selection * Octree-based AMR codes (RAMSES and ART) support slices, projections and cutting planes

If any of you are interested in helping with this reengineering effort, I would be extremely interested in collaborating. My goal for this effort is to focus on making yt a future-compatible code, and changing it from an organically grown system into one that we design based on the years of experience we all have now had with it.

-Matt

---

yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org

_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org

On Tue, Aug 7, 2012 at 11:59 AM, Samuel W Skillman wrote:

Matt,

On Tue, Aug 7, 2012 at 9:55 AM, Matthew Turk wrote:

...

Sam,

On Tue, Aug 7, 2012 at 11:51 AM, Sam Skillman wrote:

...

My goals for 2.5:

Re-written, modularized, amr_kdtree. This will allow for kd-tree decomp of generalized collections of grids as well as future optimization, as well as parallelization over arbitrary N processors, not just power of 2! Alpha channel integration for the rendering, allowing for easier control of opaque and transparent features in the same image.

This sounds really awesome.

...

Neither of these necessarily belong in the 3.0 development IMO, but I could be convinced to work in that sandbox. Both of these are working in an alpha state right now.

I agree, these don't *need* to be in 3.0, so let's wait till that branch is feature complete before I start selling you on moving the dev there. :)

Sounds good.

...

...but ... do you think the walk and the decomp could be separated, so we could feed it an octree? In the 3.0 branch I've got an oct geometry handler which could potentially be instrumented with the necessary additional info to allow fast traversal...

Yes. I think it should be straightforward to write a modified decomp for octree that retains the reduction methods for combining the final image. Then we can just pass each unit of octree to the fast traversal, whatever that ends up being.

Maybe this is what you are already thinking -- but perhaps restructuring the way the camera is setup could follow something like: 1) Decompose volume based on data source 2) Setup image plane 3) Traverse volume And that step 1 could call a method on the geometry handler, which would dispatch differently for the Octs versus for the Patch refinement.

...

-Matt

...

Sam

On Tue, Aug 7, 2012 at 9:46 AM, Matthew Turk wrote:

...

Hi all,

Now that 2.4 is out, I wanted to update you on where this roadmap (from early June, two months ago today!) is. It slipped a bit, but I think we're still broadly on track.

My current plan is to focus on development of the 3.0 branch/fork, which is located here:

http://bitbucket.org/yt_analysis/yt-3.0

It is nearly functional for all major functions from the patch-based perspective, but I am still working on projections and a handful of other operations -- flushing data back to grids (for clump finding), masking of cells (for extracted regions), and a few other medium-level features. Once it has reached feature completeness for patch-based, I think it will be ready to have many of you start testing it, kicking the tires, maybe even moving your development there. I'll update this group as I make progress, but I'd also be very appreciative if anyone wants to start testing it with me and working on it. I'm quite excited about some aspects of it, and I've already been pleased with simplifications in several areas.

We should still probably leave open the possibility of a 2.5 release, but with a much shorter changelog than 2.4 had! :)

I'll be around in IRC as well as on the mailing list if anyone has any concerns, suggestions, or wants to get involved in this.

-Matt

On Thu, Jun 7, 2012 at 4:50 PM, Matthew Turk wrote:

...

Hi all,

Sam and I are working on issuing a PR about the volume rendering refactor. This is the last major code change to go in place for 2.4, pending the quad_proj changes being accepted or rejected. I'm still thinking about this refine_by issue, and that might also make it in. After the volume_refactor, it's just testing and documentation left:

https://bitbucket.org/yt_analysis/yt/issues?status=new&status=open&milestone=2.4

After 2.4 is released, I am going to shift my new development exclusively to the yt-3.0 branch, which is currently hosted here:

http://bitbucket.org/yt_analysis/yt-3.0

This will include geometry refactoring and many remaining de-Enzoifications of the code base. 3.0 is nearing a usable state, but will not be ready for production for a while. However, I think that allowing the codebases to diverge too rapidly will lead to problems: a fracturing. So once the 3.0 codebase can muster feature parity with 2.x, I am going to ask that all development on the 2.x branch cease, and all pull requests for features and new functionality be applied only to the 3.0 branch. In advance of this, I would like to solicit help: we've identified a number of relatively straightforward changes to this code base that can and should be made. For instance, the changing of field names. I'd also like to consider moving to a more spread-out source code layout, like one class per file. Changes like these make it difficult to do cross-branch merges and transplants, which is why I would like to save them for after feature parity has been reached and development on 2.x has halted.

Previous threads:

http://lists.spacepope.org/pipermail/yt-dev-spacepope.org/2012-February/0018...

http://lists.spacepope.org/pipermail/yt-dev-spacepope.org/2012-February/0018...

http://lists.spacepope.org/pipermail/yt-dev-spacepope.org/2012-February/0018...

http://lists.spacepope.org/pipermail/yt-dev-spacepope.org/2012-March/001941....

http://lists.spacepope.org/pipermail/yt-dev-spacepope.org/2012-March/001949....

If anyone has any strong objections to this strategy, please reply.

Estimated Timeline:

* June 26, 2.4 release * July 30, 3.0 reaches rough feature parity * Fall sometime: 3.0 merged into main 'yt' branch * Some time after that: 3.0 released

Criteria for Feature Parity:

* All patch-based AMR codes support data region selection * Parallelism is functional for all parallel-capable routines * All patch-based AMR codes support projections, slices, cutting planes * All patch-based AMR codes support volume rendering * All patch-based AMR codes support halo finding * Octree-based AMR codes (RAMSES and ART) support data region selection * Octree-based AMR codes (RAMSES and ART) support slices, projections and cutting planes

If any of you are interested in helping with this reengineering effort, I would be extremely interested in collaborating. My goal for this effort is to focus on making yt a future-compatible code, and changing it from an organically grown system into one that we design based on the years of experience we all have now had with it.

-Matt

---

yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org

_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org

_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org

-- Samuel W. Skillman DOE Computational Science Graduate Fellow Center for Astrophysics and Space Astronomy University of Colorado at Boulder samuel.skillman[at]colorado.edu

_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org