Mailman 3 July 2012 - yt-dev

Near- and mid-term roadmap: 2.4, 2.x, 3.0
by Matthew Turk 07 Aug '12

07 Aug '12

Hi all, Sam and I are working on issuing a PR about the volume rendering refactor. This is the last major code change to go in place for 2.4, pending the quad_proj changes being accepted or rejected. I'm still thinking about this refine_by issue, and that might also make it in. After the volume_refactor, it's just testing and documentation left: https://bitbucket.org/yt_analysis/yt/issues?status=new&status=open&mileston… After 2.4 is released, I am going to shift my new development exclusively to the yt-3.0 branch, which is currently hosted here: http://bitbucket.org/yt_analysis/yt-3.0 This will include geometry refactoring and many remaining de-Enzoifications of the code base. 3.0 is nearing a usable state, but will not be ready for production for a while. However, I think that allowing the codebases to diverge too rapidly will lead to problems: a fracturing. So once the 3.0 codebase can muster feature parity with 2.x, I am going to ask that all development on the 2.x branch cease, and all pull requests for features and new functionality be applied only to the 3.0 branch. In advance of this, I would like to solicit help: we've identified a number of relatively straightforward changes to this code base that can and should be made. For instance, the changing of field names. I'd also like to consider moving to a more spread-out source code layout, like one class per file. Changes like these make it difficult to do cross-branch merges and transplants, which is why I would like to save them for after feature parity has been reached and development on 2.x has halted. Previous threads: http://lists.spacepope.org/pipermail/yt-dev-spacepope.org/2012-February/001… http://lists.spacepope.org/pipermail/yt-dev-spacepope.org/2012-February/001… http://lists.spacepope.org/pipermail/yt-dev-spacepope.org/2012-February/001… http://lists.spacepope.org/pipermail/yt-dev-spacepope.org/2012-March/001941… http://lists.spacepope.org/pipermail/yt-dev-spacepope.org/2012-March/001949… If anyone has any strong objections to this strategy, please reply. Estimated Timeline: * June 26, 2.4 release * July 30, 3.0 reaches rough feature parity * Fall sometime: 3.0 merged into main 'yt' branch * Some time after that: 3.0 released Criteria for Feature Parity: * All patch-based AMR codes support data region selection * Parallelism is functional for all parallel-capable routines * All patch-based AMR codes support projections, slices, cutting planes * All patch-based AMR codes support volume rendering * All patch-based AMR codes support halo finding * Octree-based AMR codes (RAMSES and ART) support data region selection * Octree-based AMR codes (RAMSES and ART) support slices, projections and cutting planes If any of you are interested in helping with this reengineering effort, I would be extremely interested in collaborating. My goal for this effort is to focus on making yt a future-compatible code, and changing it from an organically grown system into one that we design based on the years of experience we all have now had with it. -Matt

4 6

Docs and 2.4 roundup (help!)
by Matthew Turk 30 Jul '12

30 Jul '12

Hi all, There's been a flurry of activity on the docs this week. * All the cookbook recipes have been moved into the docs and the cookbook shut down. We now have 45 cookbook recipes, and they're integrated much better into the docs. * Most of the docs have been rewritten to remove PlotCollection * The data index is now part of the website repo. It's up, but the files are still copying. Here's my current draft of the changelog for 2.4. Did I forget anything? https://bitbucket.org/MatthewTurk/yt-doc/src/b57d3621ff70/source/changelog.… We're just about good to go. These things still need to be done: * Rewrite the remaining docs that use PlotCollection. Specifically, orientation/making_plots.rst needs to be completely redone. Can someone volunteer to rewrite this document? * Volume rendering docs need to be rewritten en masse, as they are quite out of date and don't reflect best practices or new functionality (ticket #382). * Do we want to put answer testing docs in for this release, or should we push that off to a point release at a slightly later date? * Write docs about the Hub, which I will do. * Check to make sure that everything that SHOULD be documented in the API docs IS documented. In the past we've also talked about slides demonstrating new functionality, as well as the possibility of doing screencasts. I'm going to throw together some HTML slides, but if anybody wants to do a screencast that would be outstanding. We actually have gotten *many* views in the past of screencasts. I think that's about it. There are two additional technical tickets that should be addressed: https://bitbucket.org/yt_analysis/yt/issue/401/entropy-field-does-not-work-… https://bitbucket.org/yt_analysis/yt/issue/390/transfer-function-bounds I can do the second, but I don't know what to do about the first. Thanks all, -Matt

6 9

Finishing up the 2.4 release
by Nathan Goldbaum 30 Jul '12

30 Jul '12

Hey everybody, I hope you're all enjoying your weekend and don't mind taking some time out of it to think about yt. Right now there are only three open tickets for the 2.4 release: 1. The entropy field in non-enzo data 2. Transfer function bounds 3. Volume rendering docs I think these tickets are things that are fixable on a short timescale. Sam, please let us know if you need more time or help with the volume rendering docs. We've put a lot of time into updating the cookbook and making sure the scripts produce good-looking images based on a variety of datasets. The latest version of the docs is available here: http://yt-project.org/docs/2.4/ I'm curious how everyone feels about setting a time aside sometime this week to do the official release. I think the release will attract more eyeballs if we can point to screencasts about some of the new functionality, including (but not limited to, see the changelog <http://yt-project.org/docs/2.4/changelog.html#version-2-4> for the full [really extensive and impressive] list of changes): 1. Threaded volume renderer 2. Plot window 3. Improved time series analysis 4. Improved extjs4 reason 5. The new yt hub I'm planning on doing a plot window screencast this afternoon. Does anyone else want to claim one of the features? Lastly, I contacted Kelle Cruz at astrobetter about going a guest post about the yt 2.4 release. She is interested - all she needs from us is a google doc with a draft of the post, including text embedded videos, and urls. I'm going to begin the write up sometime this week and then share the google doc on the yt-dev list so that others can contribute. The end is in sight! This will be by far the best release of yt yet :) Cheers, Nathan

4 8

Error in yt build from latest checkout
by Eric Hallman 25 Jul '12

25 Jul '12

I am getting a cython error in the latest checkout of yt. parent: 6116:e62a6e4015c5 tip Fixing scaling of data->pixels in the callbacks. branch: yt I pulled changes and did setup.py build, and I get this. True with a clean build as well.: building extension "yt.utilities.lib.grid_traversal" sources cythonc:> build/src.macosx-10.4-x86_64-2.7/yt/utilities/lib/grid_traversal.c Error compiling Cython file: ------------------------------------------------------------ ... if im.vd_strides[0] == -1: with nogil, parallel(num_threads = num_threads): idata = <ImageAccumulator *> malloc(sizeof(ImageAccumulator)) idata.supp_data = self.supp_data v_pos = <np.float64_t *> malloc(3 * sizeof(np.float64_t)) for j in prange(size, schedule="static",chunksize=1): ^ ------------------------------------------------------------ yt/utilities/lib/grid_traversal.pyx:368:31: Invalid keyword argument: chunksize Error compiling Cython file: ------------------------------------------------------------ ... idata.supp_data = self.supp_data v_pos = <np.float64_t *> malloc(3 * sizeof(np.float64_t)) v_dir = <np.float64_t *> malloc(3 * sizeof(np.float64_t)) # If we do not have a simple image plane, we have to cast all # our rays for j in prange(size, schedule="dynamic", chunksize=100): ^ ------------------------------------------------------------ yt/utilities/lib/grid_traversal.pyx:393:31: Invalid keyword argument: chunksize Error compiling Cython file: ------------------------------------------------------------ ... cdef np.float64_t px, py cdef np.float64_t width[3] for i in range(3): width[i] = self.width[i] if im.vd_strides[0] == -1: with nogil, parallel(num_threads = num_threads): ^ ------------------------------------------------------------ yt/utilities/lib/grid_traversal.pyx:364:32: Invalid value for num_threads argument, expected an int error: 3 errors while compiling 'yt/utilities/lib/grid_traversal.pyx' with Cython -- Eric Hallman Tech-X Corporation hallman(a)txcorp.com 5621 Arapahoe Ave, Suite A Phone: (720) 254-5833 Boulder, CO 80303 Fax: (303) 448-7756 --

2 2

Data downloader
by Matthew Turk 24 Jul '12

24 Jul '12

Hi all, For the workshop we used a download.py script written by Stephen to get all of the data, which was tarred up and put into subdirectories. John ZuHone and I have been trying to add more sample data, so that people can (if they like) get started right away with using yt. All of the cookbook recipes have been or are being rewritten to use larger datasets than the old RD0005 item, and many are being switched to use FLASH data instead of Enzo data. All in all, we're trying to push pretty hard to make the cookbook more useful. One question that's come up is: do we want to continue using download.py? The burden on uploaders is moderately higher, in that the files all have to be tarred up in a particular way, and they have to be added to download.py, but it does provide a measure of robustness. If we do this, can we move download.py into the main distribution, under scripts/ ? Stephen, John, others who have used the script, what do you think about this? -Matt

6 10

Fwd: [yt_analysis/yt] Modernized LightCone and LightRay (pull request #214)
by Matthew Turk 24 Jul '12

24 Jul '12

Britton has just submitted a pull request to bring LightCone and LightRay up to date with the current parallelism strategy. He asks a few questions that others on this list may have thoughts on; I will be leaving mine in a comment on the PR. Thanks, Britton! ---------- Forwarded message ---------- From: Britton Smith <pullrequests-noreply(a)bitbucket.org> Date: Mon, Jul 23, 2012 at 11:01 PM Subject: [yt_analysis/yt] Modernized LightCone and LightRay (pull request #214) To: matthewturk(a)gmail.com A new pull request has been opened by Britton Smith. brittonsmith/yt has changes to be pulled into yt_analysis/yt. https://bitbucket.org/yt_analysis/yt/pull-request/214/modernized-lightcone-… Title: Modernized LightCone and LightRay Major changes to the infrastructure supporting the LightCone and LightRay: 1) The LightCone and LightRay generators have now been switched over to the new SimulationTimeSeries so that they can now, in principle be used with other simulation codes. This will require other frontends to add code similar to what's yt/frontends/enzo/simulation_handling.py. 2) Both the LightCone and the LightRay are now parallelized using the parallel_objects function, so they can be used much more flexibly with multi-level parallelism. This also removes the direct mpi4py imports from LightRay. 3) LightCone and LightRay, along with there supporting superclass CosmologySplice, have been moved to yt/analysis_modules/cosmological_observation. This code has been heavily tested and retains all pre-existing functionality, plus a few more things that will be mentioned in the documentation and cookbook recipes that will be updated shortly. Before accepting this PR: 1. Are people happy with the new location: yt/analysis_modules/cosmological_observation? I believe that all of this code belongs in a subdirectory together but that was the best name I could come up with. 2. The old EnzoSimulation (yt/analysis_modules/simulation_handling) class still exists, but it is no longer a dependency for anything, and is totally replaced with the new EnzoSimulation in yt/frontends/enzo, which is totally superior. Can I remove this now or should it be deprecated and scheduled for removal at a later date? Changes to be pulled: cd09f872c174 by Britton Smith: "Merged." f7bc036677fd by Britton Smith: "Merged." a303d6f14a33 by Britton Smith: "Changing cosmology subdirectory to cosmological_observation and adding imports t…" b3c9210bf072 by Britton Smith: "Added LightRay to api file." b764002ef788 by Britton Smith: "Modernized the light ray generator and moved it to a new home. The direct mpi4py…" 41f4117f4613 by Britton Smith: "Moving light ray to the cosmology subheading." 9604d5d6187a by Britton Smith: "Adding setup for cosmology module." 6475d956eec7 by Britton Smith: "Removing light cone generator from setup." 304187324667 by Britton Smith: "Removing old light cone generator." c9a318592c97 by Britton Smith: "Cleaned up unique light cone generator." 29aa8b84169c by Britton Smith: "Adding keyword to halo mask to specify the virial overdensity to be used to calc…" bdf0cc5fa462 by Britton Smith: "Changing some more variable names." f7b9c8466b13 by Britton Smith: "Parallelized light cone halo mask with parallel objects and simplified code cons…" 042624ea193e by Britton Smith: "Continued cleanup of light cone halo mask and changed light cone variable to low…" 68472c735864 by Britton Smith: "Fixed some style conventions in light cone and began fixing light cone halo mask…" 92a989d572f3 by Britton Smith: "Switched lightcone generator from being a subclass of TimeSeriesData to using pa…" 64e5330fd707 by Britton Smith: "Merged." c7c2fcaa10b1 by Britton Smith: "Changed EnzoSimulation.get_time_series to allow further selection with find_outp…" 1125ca16272c by Britton Smith: "Adding files for new light cone into analysis_modules/cosmology. There is curren…" 5656cf35de29 by Britton Smith: "Removed enzoisms from cosmology splice." db5ec44e0121 by Britton Smith: "Changed absorption spectrum generator to increase the wavelength window until th…" 1baa32f77c80 by Britton Smith: "Fixed bug in voigt profile generator that was causing damping wings to only be a…" be28c6705909 by Britton Smith: "Adding files for CosmologyTimeSeries." 7acc4cace811 by Britton Smith: "Changed EnzoSimulation.get_time_series to allow further selection with find_outp…" -- This is an issue notification from bitbucket.org. You are receiving this either because you are the participating in a pull request, or you are following it.

1 0

Re: [yt-dev] [yt_analysis/yt-doc] Fixing sphinx errors. (pull request #37)
by Nathan Goldbaum 23 Jul '12

23 Jul '12

Hi all, This PR fixes a number of issues with the plot window. Drop by IRC and let us know if you catch any more issues. Hopefully we have most of the corner cases covered by now :) There are also some docs on the plot window in the yt-doc repo under visualization/plots.rst. Cheers, Nathan On 7/23/12 12:00 PM, Matthew Turk wrote: > New comment on pull request: > > https://bitbucket.org/yt_analysis/yt-doc/pull-request/37/fixing-sphinx-erro… > > Matthew Turk (MatthewTurk) said: > > Looks good with latest update. Thanks! > > -- > This is a pull request comment notification from bitbucket.org. > You are receiving this either because you are participating > in a pull request, or you are following it.

1 0

shpinx errors
by Nathan Goldbaum 17 Jul '12

17 Jul '12

Hi all, I've managed to fix most of the sphinx errors our docs build produces. I still get the a few warnings, but I think we can safely ignore most of them. There are still a few errors that touch parts of the docs I didn't write, specifically: yt-doc/source/advanced/creating_frontend.rst:: WARNING: document isn't included in any toctree yt-doc/source/advanced/reason_architecture.rst:: WARNING: document isn't included in any toctree yt-doc/source/visualizing/simulated_observations.rst:: WARNING: document isn't included in any toctree yt-doc/source/workshop.rst:: WARNING: document isn't included in any toctree The errors imply that these four files aren't accessible by navigating through the docs. Does anyone know how should these sections be integrated into the rest of the docs? I also get several errors complaining about missing images from the cookbook. Are the images not included in the repository for a reason? Is there a helper script somewhere to generate the images? If so, I'll look into integrating it into the build process. -Nathan

3 2

Some questions about TimeSeries
by John ZuHone 16 Jul '12

16 Jul '12

Hi all, I've been playing around with some TimeSeries examples for the cookbook and I have noticed a couple of things that seem frankly annoying and I just wanted to ping the list and see if there was a reason for them. 1) If I run more than one task in a script, it has to load all of the pfs again. Is this by design? 2) Secondly, I have noticed that what gets returned in a parameter lookup or a quantities computation has one dimension more than desired. For example, getting the simulation time of each pf in the TimeSeries yields: times = ts.params.current_time where times is a list of lists, each sublist with one member, with na.array(times).shape == (number of pfs, 1) The same sort of thing happens with returning, say, the angular momentum vector, which will have a shape of (number of pfs, 1, 3). It seems to me that what should be returned is a NumPy array with the dimensionality scaled down by one, e.g. the lists "in the middle" should be eliminated. Best, John Z

3 6

API links in the docs?
by Nathan Goldbaum 16 Jul '12

16 Jul '12

Hi all, It seems that the recent changes to the docs have broken the internal links to api references. Since the docstrings are inside the api references, this means the docstrings don't show up in new builds of the docs. I'm not sure if this is a sphinx issue or an issue with the tip of yt-doc. I think there was some discussion about consolidating the api reference pages, which I am in favor of, but we should still make sure the docstrings are available inside the html docs. Nathan

2 7