Hi guys, I'd like to take a moment to sketch out my proposal for releasing yt-1.5. We all run off trunk, but right now, 1.0 is the version included with Enzo. Furthermore, the documentation has not been substantially updated since 1.0, so almost all of the stuff we have been doing and working on for the past year (yes, *year*) has not been advertised or included. I'd like to compile some of the things that have happened, most of which have been visible in the ticket tracker, but not all. http://yt.enzotools.org/query?status=closed&group=resolution&milestone=1.5 There are a couple things that don't show up quite so visibly: * Fully parallel slices, projections, cutting planes, profiles, quantities (although scaling will be a target of 2.0) * Mostly parallel HOP (scaling is a major problem here) * FoF halo finding * Object storage and serialization * Major performance improvements to the clump finder (factor ~5) * Generalized domain sizes * Generalized field info containers (which need some work for 2.0) * Dark Matter-only simulations * 1D and 2D simulations * Better IO for HDF5 sets * Support for the Orion AMR code * Spherical re-gridding * Halo profiler (ticket 95 needs some love) * Light cone generator * Callback interface improved * Several new callbacks * New data objects -- ortho and non-ortho rays, limited ray-tracing * Fixed resolution buffers * Spectral integrator for CLOUDY data * Substantially better interactive interface * Performance improvements basically everywhere * Command-line interface to *many* common tasks * Isolated plot handling, independent of PlotCollections I'm pretty sure there's a lot more I'm forgetting -- but the improvements have been numerous. I think the biggest thing to emphasize here would be that yt works in parallel. Scaling is not ideal. That's going to be a task for a later date, I do believe, and for now I think it's important to push out the improvements -- frankly, I'm a little embarrassed people may be using yt-1.0 at this point, because so much work has gone on in the trunk to improve basically every aspect of the code. (And yes, our audience is small, by almost all metrics: but I think this particular set of improvements will help to increase it.) I think this release is much more compelling than the last, and I think we all have something to be proud of. We still have to finish up these tickets: http://yt.enzotools.org/report/12 Most or all of these are my repsonsibility; I'd like some help with #95, though. Additionally, if anybody wants to volunteer to look at (even if just attempting to replicate and getting proper tracebacks) #189, #191 and #197, I'd really appreciate it. #208 might get bumped to 2.0, since I'm the only one using fido anymore; I'd like to bring it back in some way some day, but right now it's not a priority. Now, for the meat of the remaining process: the documentation. This will be primarily my responsibility, but I will need help from other people. Specifically, I need someone to volunteer to read and edit what I write. Additionally, I would like to submit a request that sections of prose be written, in the style of a tutorial, by the people who really are the experts on different sections of the code. I've noticed a substantial uptick in feelings of ownership of various sections, and I'm really happy about that. I think we really need some semi-tutorial style sections written by the people who feel they know best certain sections of the code that are really valuable and useful. Here's my proposed set: * Halo finding: Stephen * Halo profiler: Britton * Clump finder: Britton * Light Cone Generator: Britton * Interactive Usage: Jeff Now, I know you are all very busy -- me too! -- so if this doesn't work for you, it's all good, just let me know. :) I'll handle all the format conversions and integration and so forth. The current documentation repo (has not been put in SVN, but will be once it's working better) is here: http://hg.enzotools.org/yt-doc/ . Furthermore, there's a page set up for comments on the existing documentation: http://yt.enzotools.org/wiki/Plans/DocEnhancements Please, if you have any complaints about existing documentation, making them there will ensure that while I go through the process of editing and rewriting, I will ensure they are taken into account. Additionally, while I am willing to take the burden of documentation, I would really love it if somebody else would help out. So, let me know if you are interested. Okay guys, thanks so much for trudging through this email. Any thoughts on any of this? Did I forget any big features? Anybody feel like helping with extra documentation? Anything else that needs to go in? We've all put a lot of work into this stuff, and I think we have a great chance of making a big splash with a new release. Thanks guys, and talk to you soon, -Matt
Hi Matt,
Most or all of these are my repsonsibility; I'd like some help with #95, though. Additionally, if anybody wants to volunteer to look at (even if just attempting to replicate and getting proper tracebacks) #189, #191 and #197, I'd really appreciate it. #208 might get bumped to 2.0, since I'm the only one using fido anymore; I'd like to bring it back in some way some day, but right now it's not a priority.
If I get a chance I'll see if I can help at all with those tickets, but no promises. I've never used reason so I don't know how much of a help I can be.
* Halo finding: Stephen
This is no problem. I'll be happy to contribute that! _______________________________________________________ sskory@physics.ucsd.edu o__ Stephen Skory http://physics.ucsd.edu/~sskory/ _.>/ _Graduate Student ________________________________(_)_\(_)_______________
Ticket 95 is mine, and I'll take all my documentation assignments. On Tue, Apr 21, 2009 at 12:41 PM, Stephen Skory <stephenskory@yahoo.com>wrote:
Hi Matt,
Most or all of these are my repsonsibility; I'd like some help with #95, though. Additionally, if anybody wants to volunteer to look at (even if just attempting to replicate and getting proper tracebacks) #189, #191 and #197, I'd really appreciate it. #208 might get bumped to 2.0, since I'm the only one using fido anymore; I'd like to bring it back in some way some day, but right now it's not a priority.
If I get a chance I'll see if I can help at all with those tickets, but no promises. I've never used reason so I don't know how much of a help I can be.
* Halo finding: Stephen
This is no problem. I'll be happy to contribute that!
_______________________________________________________ sskory@physics.ucsd.edu o__ Stephen Skory http://physics.ucsd.edu/~sskory/ <http://physics.ucsd.edu/%7Esskory/> _.>/ _Graduate Student ________________________________(_)_\(_)_______________ _______________________________________________ Yt-dev mailing list Yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
Hi guys, Great to hear it! I think it's not unreasonable to estimate a release on May 15, when I'll submit to PyPi, email the enzo list, and hopefully (this will be cleared first with the folks on enzo-l) switch the public enzo repo to a 1.5 branch rather than the 1.0 branch. That's nearly a month from now, and if we get the docs and tickets closed by then, we're ready to go. I'm going to start performing surgery on the (in hg) docs later tonight; I'm also going to do my best to keep a build of them up to date somewhere. I'll let you know when I figure out where that will be. -Matt On Tue, Apr 21, 2009 at 12:39 PM, Britton Smith <brittonsmith@gmail.com> wrote:
Ticket 95 is mine, and I'll take all my documentation assignments.
On Tue, Apr 21, 2009 at 12:41 PM, Stephen Skory <stephenskory@yahoo.com> wrote:
Hi Matt,
Most or all of these are my repsonsibility; I'd like some help with #95, though. Additionally, if anybody wants to volunteer to look at (even if just attempting to replicate and getting proper tracebacks) #189, #191 and #197, I'd really appreciate it. #208 might get bumped to 2.0, since I'm the only one using fido anymore; I'd like to bring it back in some way some day, but right now it's not a priority.
If I get a chance I'll see if I can help at all with those tickets, but no promises. I've never used reason so I don't know how much of a help I can be.
* Halo finding: Stephen
This is no problem. I'll be happy to contribute that!
_______________________________________________________ sskory@physics.ucsd.edu o__ Stephen Skory http://physics.ucsd.edu/~sskory/ _.>/ _Graduate Student ________________________________(_)_\(_)_______________ _______________________________________________ Yt-dev mailing list Yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
_______________________________________________ Yt-dev mailing list Yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
Hi all, I think it's time to make the parallel HOP interface default and remove SS_HopOutput. I'm writing up the halo finding docs right now and I realize that it's time. Parallel HOP is now trusted and superior to the old way of doing things. However, now that I added a FoF finder into yt, maybe it shouldn't be called HaloFinder? Agreed? _______________________________________________________ sskory@physics.ucsd.edu o__ Stephen Skory http://physics.ucsd.edu/~sskory/ _.>/ _Graduate Student ________________________________(_)_\(_)_______________
I agree, it should be renamed and the new version should supplant the old. It seems to me that the best course of action would be severalfold, and committed in a single go. 1. Join the yt/lagos/fof/FOF_Output.py and yt/lagos/hop/HOP_Output.py files and put them as yt/lagos/HaloFinder.py 2. Remove as much code duplication as possible between the two by refactoring into a base class and two subclasses 3. Add HOPHaloFinder and FOFHaloFinder classes 4. set HaloFinder = HOPHaloFinder at the bottom of yt/lagos/HaloFinder.py 5. Change all the recipes in examples/ and on the Wiki to use the new API 6. Fix yt/mods.py to import from the correct location 7. Email yt-users to notify about the change in API, but don't stress the parallel nature just yet. We can do this in a few commits if necessary, but it looks like it could all be done as a single one. I can help out with #2 after the initial commit and change, but I think you have a better feel for how it should go at the start. It'd be really great if you could test this before committing, too, in both serial and parallel. I don't mind breaking the API just now, but I'd prefer we did it in a single go. This is great news! Thanks for your hard work. -Matt On Wed, Apr 22, 2009 at 9:22 AM, Stephen Skory <stephenskory@yahoo.com> wrote:
Hi all,
I think it's time to make the parallel HOP interface default and remove SS_HopOutput. I'm writing up the halo finding docs right now and I realize that it's time. Parallel HOP is now trusted and superior to the old way of doing things. However, now that I added a FoF finder into yt, maybe it shouldn't be called HaloFinder?
Agreed?
_______________________________________________________ sskory@physics.ucsd.edu o__ Stephen Skory http://physics.ucsd.edu/~sskory/ _.>/ _Graduate Student ________________________________(_)_\(_)_______________ _______________________________________________ Yt-dev mailing list Yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
1. Join the yt/lagos/fof/FOF_Output.py and yt/lagos/hop/HOP_Output.py files and put them as yt/lagos/HaloFinder.py 2. Remove as much code duplication as possible between the two by refactoring into a base class and two subclasses
I'm actually kind of hesitant to do this. FoF works, but it's still what I'd call 'experimental.' I haven't done enough testing to trust it yet. I haven't even done a comparison suite with HOP. I think it's OK to put it out there and let people use it, it's not flaky in that sense. In my documentation, I'll write in bold letters something like 'FoF is mostly untested, so if you use it, check your results to make sure they're reasonable!' Although the chances are small, I don't want to merge together something trusted with something untrusted. Do you see what I mean? _______________________________________________________ sskory@physics.ucsd.edu o__ Stephen Skory http://physics.ucsd.edu/~sskory/ _.>/ _Graduate Student ________________________________(_)_\(_)_______________
I understand; but what I am suggesting is that, from a diff of FOF_Output and SS_HopOutput, they are largely identical. I don't think this is awesome; I agree that HOP should be the default, but what I'd like to do is consolidate the code base. If you'd rather not explicitly import FOFHaloFinder in yt/mods.py I understand, but as long as we're going to be working on this stuff in the near-term future I think we should make it the best we can. That being said, I greatly appreciate your reluctance to break known entities. But I don't think we have to -- I'd just like to see them share a codebase, rather than a duplicated one! -Matt On Wed, Apr 22, 2009 at 11:13 AM, Stephen Skory <stephenskory@yahoo.com> wrote:
1. Join the yt/lagos/fof/FOF_Output.py and yt/lagos/hop/HOP_Output.py files and put them as yt/lagos/HaloFinder.py 2. Remove as much code duplication as possible between the two by refactoring into a base class and two subclasses
I'm actually kind of hesitant to do this. FoF works, but it's still what I'd call 'experimental.' I haven't done enough testing to trust it yet. I haven't even done a comparison suite with HOP. I think it's OK to put it out there and let people use it, it's not flaky in that sense. In my documentation, I'll write in bold letters something like 'FoF is mostly untested, so if you use it, check your results to make sure they're reasonable!'
Although the chances are small, I don't want to merge together something trusted with something untrusted. Do you see what I mean? _______________________________________________________ sskory@physics.ucsd.edu o__ Stephen Skory http://physics.ucsd.edu/~sskory/ _.>/ _Graduate Student ________________________________(_)_\(_)_______________ _______________________________________________ Yt-dev mailing list Yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
Matt,
That being said, I greatly appreciate your reluctance to break known entities. But I don't think we have to -- I'd just like to see them share a codebase, rather than a duplicated one!
Is this what you're thinking about? http://paste.enzotools.org/show/106/ I've done a quick merge of the two files together. The differences (mainly where HOP deals with max dens stuff) have "if HOP:" and "if FoF" in front of them, which isn't functional right now but that's easily remedied. There are a couple 'if FOF's there too. _______________________________________________________ sskory@physics.ucsd.edu o__ Stephen Skory http://physics.ucsd.edu/~sskory/ _.>/ _Graduate Student ________________________________(_)_\(_)_______________
Hi Stephen, Not really... I've taken a whack at it, but I have not tested the Friends of Friends terribly well, as you suggested that was not a priority. It worked in both parallel and serial, however. I'd like to note that it seems to me there is a problem with the density output from the parallel hop; it gives different answers based on the tiling. I tested this against trunk and got the same thing. I feel like we had a fix for this that may have gotten lost. Here's my initial proposal for the combined halo finder. I've set densities in the FOF output to be -1 because I feel it is important to preserve the output between the two different formats, as they may be used as input to a unified halo profiler. http://paste.enzotools.org/show/107/ One of my test scripts can be seen here: http://hg.enzotools.org/test_scripts/file/tip/test_halo_finder.py and downloaded here: http://hg.enzotools.org/test_scripts/raw-file/17789811f400/test_halo_finder.... -Matt On Wed, Apr 22, 2009 at 12:33 PM, Stephen Skory <stephenskory@yahoo.com> wrote:
Matt,
That being said, I greatly appreciate your reluctance to break known entities. But I don't think we have to -- I'd just like to see them share a codebase, rather than a duplicated one!
Is this what you're thinking about?
http://paste.enzotools.org/show/106/
I've done a quick merge of the two files together. The differences (mainly where HOP deals with max dens stuff) have "if HOP:" and "if FoF" in front of them, which isn't functional right now but that's easily remedied.
There are a couple 'if FOF's there too.
_______________________________________________________ sskory@physics.ucsd.edu o__ Stephen Skory http://physics.ucsd.edu/~sskory/ _.>/ _Graduate Student ________________________________(_)_\(_)_______________ _______________________________________________ Yt-dev mailing list Yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
Matt,
Not really... I've taken a whack at it, but I have not tested the Friends of Friends terribly well, as you suggested that was not a priority. It worked in both parallel and serial, however.
Excellent. I'm sorry I didn't know what you meant.
I'd like to note that it seems to me there is a problem with the density output from the parallel hop; it gives different answers based on the tiling. I tested this against trunk and got the same thing. I feel like we had a fix for this that may have gotten lost.
I think it just needs to be adjusted by the same factor that the threshold is being corrected by. Perhaps by the inverse? _______________________________________________________ sskory@physics.ucsd.edu o__ Stephen Skory http://physics.ucsd.edu/~sskory/ _.>/ _Graduate Student ________________________________(_)_\(_)_______________
Excellent. I'm sorry I didn't know what you meant.
No prob, sorry if I sounded gruff, didn't mean to be. I appreciate your work on this. If you could test it out that would be great, then we can commit it.
I think it just needs to be adjusted by the same factor that the threshold is being corrected by. Perhaps by the inverse?
Can you check this and implement it in the new file? -Matt
Matt,
Can you check this and implement it in the new file?
I ran some tests and it looks good, for both HOP and FoF. I think I've also fixed the maximum overdensity value, too. See: http://paste.enzotools.org/compare/108/107/ The values between serial and parallel hop with this fix are the same to 1 part in 1,000,000. _______________________________________________________ sskory@physics.ucsd.edu o__ Stephen Skory http://physics.ucsd.edu/~sskory/ _.>/ _Graduate Student ________________________________(_)_\(_)_______________
Awesome, looks great! So I guess if we go back to the list I suggested: 1. Join the yt/lagos/fof/FOF_Output.py and yt/lagos/hop/HOP_Output.py files and put them as yt/lagos/HaloFinder.py 2. Remove as much code duplication as possible between the two by refactoring into a base class and two subclasses 3. Add HOPHaloFinder and FOFHaloFinder classes 4. set HaloFinder = HOPHaloFinder at the bottom of yt/lagos/HaloFinder.py 5. Change all the recipes in examples/ and on the Wiki to use the new API 6. Fix yt/mods.py to import from the correct location 7. Email yt-users to notify about the change in API, but don't stress the parallel nature just yet. we've done 1, 2, 3. I think we can remove HopOutput.py, SS_HopOutput.py, and FOF_Output.py and add HaloFinder.py in one go. I can do this, which is 4 and 6, if you like and you can take care of 5 and 7. Sound good? -Matt On Wed, Apr 22, 2009 at 9:15 PM, Stephen Skory <stephenskory@yahoo.com> wrote:
Matt,
Can you check this and implement it in the new file?
I ran some tests and it looks good, for both HOP and FoF. I think I've also fixed the maximum overdensity value, too. See:
http://paste.enzotools.org/compare/108/107/
The values between serial and parallel hop with this fix are the same to 1 part in 1,000,000.
_______________________________________________________ sskory@physics.ucsd.edu o__ Stephen Skory http://physics.ucsd.edu/~sskory/ _.>/ _Graduate Student ________________________________(_)_\(_)_______________ _______________________________________________ Yt-dev mailing list Yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
Matt,
5. Change all the recipes in examples/ and on the Wiki to use the new API
7. Email yt-users to notify about the change in API, but don't stress the parallel nature just yet.
we've done 1, 2, 3. I think we can remove HopOutput.py, SS_HopOutput.py, and FOF_Output.py and add HaloFinder.py in one go. I can do this, which is 4 and 6, if you like and you can take care of 5 and 7.
I've done 5. Once you commit your stuff I'll do 7. Don't forget to commit the updated setup.py in lagos for FoF. _______________________________________________________ sskory@physics.ucsd.edu o__ Stephen Skory http://physics.ucsd.edu/~sskory/ _.>/ _Graduate Student ________________________________(_)_\(_)_______________
Done in r1274. As per the commit message and our discussion over IM: from yt.mods import * pf = load("data0001") halo_list = HaloFinder(pf) is the correct mechanism for running the halo finder. This defaults to using HOP, and will work in both parallel and serial with a default padding of 0.02. I'll email yt-users with this change. -Matt On Thu, Apr 23, 2009 at 10:02 AM, Stephen Skory <stephenskory@yahoo.com> wrote:
Matt,
5. Change all the recipes in examples/ and on the Wiki to use the new API
7. Email yt-users to notify about the change in API, but don't stress the parallel nature just yet.
we've done 1, 2, 3. I think we can remove HopOutput.py, SS_HopOutput.py, and FOF_Output.py and add HaloFinder.py in one go. I can do this, which is 4 and 6, if you like and you can take care of 5 and 7.
I've done 5. Once you commit your stuff I'll do 7. Don't forget to commit the updated setup.py in lagos for FoF.
_______________________________________________________ sskory@physics.ucsd.edu o__ Stephen Skory http://physics.ucsd.edu/~sskory/ _.>/ _Graduate Student ________________________________(_)_\(_)_______________ _______________________________________________ Yt-dev mailing list Yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
----- Original Message ----
From: Matthew Turk <matthewturk@gmail.com> To: yt-dev@lists.spacepope.org Sent: Thursday, April 23, 2009 10:06:27 AM Subject: Re: [Yt-dev] Release 1.5
Done in r1274. As per the commit message and our discussion over IM:
from yt.mods import * pf = load("data0001") halo_list = HaloFinder(pf)
I guess this needs to be changed in HaloProfiler too? _______________________________________________________ sskory@physics.ucsd.edu o__ Stephen Skory http://physics.ucsd.edu/~sskory/ _.>/ _Graduate Student ________________________________(_)_\(_)_______________
Matt,
I fetched this. Shall I put the HaloFinder docs I'm about to write in the cookbook section? I'll add any new example code into the yt/examples directory like the other stuff in the cookbook directory of the docs. Or would you prefer it on the wiki? _______________________________________________________ sskory@physics.ucsd.edu o__ Stephen Skory http://physics.ucsd.edu/~sskory/ _.>/ _Graduate Student ________________________________(_)_\(_)_______________
I think it'd be great to put them somewhere in there; the cookbook is fine for now, and I'll add you as a sectionauthor when I next update the docs. I might end up moving them around as the docs need some reworking, however, but I'll take care of that. Thanks! -Matt On Thu, Apr 23, 2009 at 10:15 AM, Stephen Skory <stephenskory@yahoo.com> wrote:
Matt,
I fetched this. Shall I put the HaloFinder docs I'm about to write in the cookbook section? I'll add any new example code into the yt/examples directory like the other stuff in the cookbook directory of the docs.
Or would you prefer it on the wiki?
_______________________________________________________ sskory@physics.ucsd.edu o__ Stephen Skory http://physics.ucsd.edu/~sskory/ _.>/ _Graduate Student ________________________________(_)_\(_)_______________ _______________________________________________ Yt-dev mailing list Yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
participants (3)
-
Britton Smith -
Matthew Turk -
Stephen Skory