Hi everyone,
We're proud to announce the third ALPHA release of yt 3.0. yt has recently transitioned to a time-based release plan ( https://ytep.readthedocs.org/en/latest/YTEPs/YTEP-0008.html ) and this is the third scheduled alpha of 3.0, although we've grossly missed the estimated date of July 15. No date for a "final" release has yet been set, but there will likely be several more alphas before that time.
The yt 2.5 codebase, and further updates in the 2.x series, will be supported for a considerable amount of time and you do not need to upgrade.
= yt 3.0?! =
yt 3.0 represents a new direction forward for yt: getting rid of all the underlying assumptions that data needs to be sectioned off into nice little grid patches. This includes supporting Octree codes natively (NMSU-ART and RAMSES), eventual support for SPH codes, and even opaque data structures where the data is extremely large (ARTIO). We're even planning support for natively handling cylindrical and spherical coordinates.
More: http://blog.yt-project.org/post/WhatsUpWith30.html
However, this is an alpha release. Not all of the existing codes have been ported to 3.0.
Additionally, this release benefits from the technical and non-technical contributions from many new people. yt is developed in the context of a community of contributors, and with the push toward a new architecture, we aim to expand that community considerably. In particular, this release has considerably benefited from contributions from many new individuals.
= Getting It! =
To try out yt 3.0, you can now pull from the main yt repository, update to the yt-3.0 branch, and rebuild your extensions. Or, if you would like to create a new, safely sectioned off environment, simply re-run the normal "development" install script after changing the variable BRANCH to "yt-3.0".
If you would like to try out yt 3.0 and are having trouble, please write to the yt-users mailing list for assistance.
The yt 3.0 install script may also work, which can be obtained by executing these commands:
wget http://hg.yt-project.org/yt/raw/yt-3.0/doc/install_script.sh bash install_script.sh
= What's New? =
A demo notebook demonstrating much new functionality, and including the full release notes, can be found here:
http://hub.yt-project.org/nb/88u14f
The bullet-pointed release notes can be found at the end of this email, as well. The main improvements include considerable memory usage reduction, ARTIO spatial data indexing support, better particle support, an overhaul of the selection methods for data objects, and a mechanism for on-the-fly definitions of particle types based on boolean filters.
Additionally, this release was used in the production of the AGORA project flagship paper. The analysis script used there can be seen at this short URL:
and the paper can be found here: http://arxiv.org/abs/1308.2669
= Reporting Problems =
If you test out yt 3.0 we want to hear if it DID or DID NOT work! Feedback is crucial at this time. yt-users and yt-dev are both good forums for discussion, asking questions, and reporting problems. Lots of things have changed on the backend, but we have attempted to minimize the user-facing changes.
To report a bug please go here:
https://bitbucket.org/yt_analysis/yt/issues/new
Note that you will not receive updates if you are not logged in when you create the bug.
= What's Next? =
The next alpha release (3.0a4) will be released sometime this fall, but development can be monitored either at http://bitbucket.org/yt_analysis/yt-3.0 or in the main yt repository under the named branch "yt-3.0". We hope to focus on generalizing Octree support further, adding better non-Cartesian support, and supporting generic smoothing kernel definitions.
If you'd like to participate in yt development, please stop by #yt on irc.freenode.net ( http://yt-project.org/irc.html ) or yt-dev ( http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org ), or submit a pull request on BitBucket.
Thanks very much,
Matt, on behalf of the yt development team
Changelog:
(Including 434 changesets from 7 contributors since 3.0a2)
- Fixes for volume rendering and porting of the Cython kD-tree. VR
now works for octree codes, albeit slowly.
- Major reorganization and simplification of selector code, including generic periodicity
- ARTIO now has spatial data support
- Major bug fix for FLASH IO. FLASH works again, following a regression in 3.0a2.
- Merged with mainline yt development, bringing developments from yt 2.5.3 and 2.5.4 into 3.0.
- OWLS (Gadget-HDF5) data now updated to match functionality of other Gadget datasets.
- Creationg of "arbitrary_grid" data selector for flexible particle deposition operations. This enables creation of a grid of arbitrary dimensions that can have particles smoothed or deposited within them.
- Conversion of all CIC operations to particle deposition operations. This results in much more flexible particle type selection and speed improvements.
- Refactoring of particle fields to enable simpler creation and collection of particle fields. Particle fields defined for Eulerian codes and Lagrangian codes are now interchangeable.
- New, currently unused parallel ring iteration. To be explored in the future for smoothing kernels.
- Many improvements for NMSU-ART
- Fixed a crazy QuadTree deposition bug for projections that showed up when projecting octree particle deposition results through a restricted data object.
- Fixed a crazy off-by-one bug for binary Gadget IO.
- Enormous Octree diet.
- Oct leaf nodes now cost 256 bits each, which may eventually be reduced to 192 bits. Previous versions were considerably larger.
- Octree traversal is strictly recursive, enabling distributed-memory octrees to be implemented.
- RAMSES octrees are created by-domain instead of globally. This will enable better parallelism in the future and faster ghost zone generation when that is implemented.
- Particle octree are now constructed via Z-curve generation. This is considerably faster (10-20x) as well as much more memory conservative. This was designed to utilize parallel octree construction in the future.
- Particle octrees are now indexed by coarse base-level indexing of domain regions. This enables unsorted particle files to be indexed for IO as well as spatial selection without mandating that leaf nodes are fully-contained within a single file. Reduction in total oct leaf count of ~8x for multi-file particle datasets.
- YTDataContainer no longer requires .shape and .size
- Only requested fields are plotted, not translated fields
- Reduce memory usage of spatially-chunked fields for patch datasets
- Added arbitrary particle loader, load_particles.
- Units fix for RAMSES boxlen!=1.0 datasets
- Particle filtering for dynamically-created particle types, similar to derived fields
- Improve type consistency in TotalQuantity and other derived quantities.
- Fixed major error with ghost zone generation and uninitialized values
- Some resiliency for particle datasets that do not specify a domain
- Bugfix for 1- and 2-D Enzo datasets
- Improvements to the Tipsy frontend, including auto-detecting parameter files
- Enable specification of n_ref when creating particle dataset
- Enable particle deposition to back-act on particle fields
- Major fix for octree particle counting, resulting in smaller meshes
- Enable "source" specification for volume rendering
- Enable initial volume rendering of octree datasets
- Fixed error with RAMSES C/F ordering of data
- Many fixes related to field dependency calculation
- FLASH cylindrical & polar pixelization fix
Hi all,
Thanks to Colin DeGraf, Chris Moody and Nathan Goldbaum for identifying and correcting a critical bug in the RAMSES data loader. This fix has now been pushed and safeguards have been put in place to prevent a regression.
Best,
Matt
On Wed, Aug 21, 2013 at 2:53 PM, Matthew Turk matthewturk@gmail.com wrote:
Hi everyone,
We're proud to announce the third ALPHA release of yt 3.0. yt has recently transitioned to a time-based release plan ( https://ytep.readthedocs.org/en/latest/YTEPs/YTEP-0008.html ) and this is the third scheduled alpha of 3.0, although we've grossly missed the estimated date of July 15. No date for a "final" release has yet been set, but there will likely be several more alphas before that time.
The yt 2.5 codebase, and further updates in the 2.x series, will be supported for a considerable amount of time and you do not need to upgrade.
= yt 3.0?! =
yt 3.0 represents a new direction forward for yt: getting rid of all the underlying assumptions that data needs to be sectioned off into nice little grid patches. This includes supporting Octree codes natively (NMSU-ART and RAMSES), eventual support for SPH codes, and even opaque data structures where the data is extremely large (ARTIO). We're even planning support for natively handling cylindrical and spherical coordinates.
More: http://blog.yt-project.org/post/WhatsUpWith30.html
However, this is an alpha release. Not all of the existing codes have been ported to 3.0.
Additionally, this release benefits from the technical and non-technical contributions from many new people. yt is developed in the context of a community of contributors, and with the push toward a new architecture, we aim to expand that community considerably. In particular, this release has considerably benefited from contributions from many new individuals.
= Getting It! =
To try out yt 3.0, you can now pull from the main yt repository, update to the yt-3.0 branch, and rebuild your extensions. Or, if you would like to create a new, safely sectioned off environment, simply re-run the normal "development" install script after changing the variable BRANCH to "yt-3.0".
If you would like to try out yt 3.0 and are having trouble, please write to the yt-users mailing list for assistance.
The yt 3.0 install script may also work, which can be obtained by executing these commands:
wget http://hg.yt-project.org/yt/raw/yt-3.0/doc/install_script.sh bash install_script.sh
= What's New? =
A demo notebook demonstrating much new functionality, and including the full release notes, can be found here:
http://hub.yt-project.org/nb/88u14f
The bullet-pointed release notes can be found at the end of this email, as well. The main improvements include considerable memory usage reduction, ARTIO spatial data indexing support, better particle support, an overhaul of the selection methods for data objects, and a mechanism for on-the-fly definitions of particle types based on boolean filters.
Additionally, this release was used in the production of the AGORA project flagship paper. The analysis script used there can be seen at this short URL:
and the paper can be found here: http://arxiv.org/abs/1308.2669
= Reporting Problems =
If you test out yt 3.0 we want to hear if it DID or DID NOT work! Feedback is crucial at this time. yt-users and yt-dev are both good forums for discussion, asking questions, and reporting problems. Lots of things have changed on the backend, but we have attempted to minimize the user-facing changes.
To report a bug please go here:
https://bitbucket.org/yt_analysis/yt/issues/new
Note that you will not receive updates if you are not logged in when you create the bug.
= What's Next? =
The next alpha release (3.0a4) will be released sometime this fall, but development can be monitored either at http://bitbucket.org/yt_analysis/yt-3.0 or in the main yt repository under the named branch "yt-3.0". We hope to focus on generalizing Octree support further, adding better non-Cartesian support, and supporting generic smoothing kernel definitions.
If you'd like to participate in yt development, please stop by #yt on irc.freenode.net ( http://yt-project.org/irc.html ) or yt-dev ( http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org ), or submit a pull request on BitBucket.
Thanks very much,
Matt, on behalf of the yt development team
Changelog:
(Including 434 changesets from 7 contributors since 3.0a2)
- Fixes for volume rendering and porting of the Cython kD-tree. VR
now works for octree codes, albeit slowly.
- Major reorganization and simplification of selector code, including generic periodicity
- ARTIO now has spatial data support
- Major bug fix for FLASH IO. FLASH works again, following a regression in 3.0a2.
- Merged with mainline yt development, bringing developments from yt 2.5.3 and 2.5.4 into 3.0.
- OWLS (Gadget-HDF5) data now updated to match functionality of other Gadget datasets.
- Creationg of "arbitrary_grid" data selector for flexible particle deposition operations. This enables creation of a grid of arbitrary dimensions that can have particles smoothed or deposited within them.
- Conversion of all CIC operations to particle deposition operations. This results in much more flexible particle type selection and speed improvements.
- Refactoring of particle fields to enable simpler creation and collection of particle fields. Particle fields defined for Eulerian codes and Lagrangian codes are now interchangeable.
- New, currently unused parallel ring iteration. To be explored in the future for smoothing kernels.
- Many improvements for NMSU-ART
- Fixed a crazy QuadTree deposition bug for projections that showed up when projecting octree particle deposition results through a restricted data object.
- Fixed a crazy off-by-one bug for binary Gadget IO.
- Enormous Octree diet.
- Oct leaf nodes now cost 256 bits each, which may eventually be reduced to 192 bits. Previous versions were considerably larger.
- Octree traversal is strictly recursive, enabling distributed-memory octrees to be implemented.
- RAMSES octrees are created by-domain instead of globally. This will enable better parallelism in the future and faster ghost zone generation when that is implemented.
- Particle octree are now constructed via Z-curve generation. This is considerably faster (10-20x) as well as much more memory conservative. This was designed to utilize parallel octree construction in the future.
- Particle octrees are now indexed by coarse base-level indexing of domain regions. This enables unsorted particle files to be indexed for IO as well as spatial selection without mandating that leaf nodes are fully-contained within a single file. Reduction in total oct leaf count of ~8x for multi-file particle datasets.
- YTDataContainer no longer requires .shape and .size
- Only requested fields are plotted, not translated fields
- Reduce memory usage of spatially-chunked fields for patch datasets
- Added arbitrary particle loader, load_particles.
- Units fix for RAMSES boxlen!=1.0 datasets
- Particle filtering for dynamically-created particle types, similar to derived fields
- Improve type consistency in TotalQuantity and other derived quantities.
- Fixed major error with ghost zone generation and uninitialized values
- Some resiliency for particle datasets that do not specify a domain
- Bugfix for 1- and 2-D Enzo datasets
- Improvements to the Tipsy frontend, including auto-detecting parameter files
- Enable specification of n_ref when creating particle dataset
- Enable particle deposition to back-act on particle fields
- Major fix for octree particle counting, resulting in smaller meshes
- Enable "source" specification for volume rendering
- Enable initial volume rendering of octree datasets
- Fixed error with RAMSES C/F ordering of data
- Many fixes related to field dependency calculation
- FLASH cylindrical & polar pixelization fix
-
Matthew Turk