Rick, I don't think he's suggesting we change enzo. I think he simply wants to know what functionality is lacking from yt's haloprofiler that enzoanyl currently has. That said, I think when we get haloprofiler working up to spec with enzoanyl's capabilities, we should announce it to the enzo list, so people who use enzoanyl (there aren't many of us) will know that there is a better supported alternative in yt. Cameron On 04/12/2011 12:18 PM, Rick Wagner wrote:

Hi Britton,

On Apr 12, 2011, at 9:04 AM, Britton Smith wrote:

...

Hey everyone,

There is a fair amount of discussion about the HaloProfiler being a replacement for enzo_anyl. I would definitely like to see this become official, since it seems to be keeping a number of people from making the switch to YT. Can somebody outline the things that enzo_anyl does that the HaloProfiler cannot? I have been under the impression that there wasn't anything, but I haven't used enzo_anyl in quite a while. I think that the HaloProfiler could easily be wrapped to look at act just like enzo_anyl, which makes this an attractive choice. But, I would post this question also the enzo-dev mailing list, since you're asking about changing Enzo, not YT.

--Rick

_______________________________________________ Yt-dev mailing list Yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org

-- Cameron Hummels PhD Candidate, Astronomy Department of Columbia University Public Outreach Director, Astronomy Department of Columbia University NASA IYA New York State Student Ambassador http://outreach.astro.columbia.edu PGP: 0x06F886E3

Hi all, I had an opportunity to speak yesterday with Greg Bryan (who's not subscribed to this list) and independent of this thread, he brought up replacing enzo_anyl with yt. The one thing that I think we may be missing is the cooling time. To get the cooling time in yt we have three options: 1) Re-calculate from scratch, using all the items in the parameter file that govern the cooling time. This will also allow it to work for non-Enzo simulation outputs. 2) Wrap the Enzo cooling time routines. This will require utilizing fortran/C interop, possibly with f2py. It also has a lot of moving parts. (These would be solved if some day there was some agreement on microphysical solver APIs.) This would work with non-Enzo datasets. 3) Mandate that if you want the cooling time in your profiles, you run with OutputCoolingTime=1 in Enzo. This would not work with non-Enzo datasets, unless they too had an option to output the local cooling time in every cell. I think #3 is the fastest time to solution, and certainly the option that is the least error prone. (The list above, it seems to me, is actually in order of increasing reliability.) The remaining items that we would need to accomplish to make HaloProfiler a generalized replacement for enzo_anyl, I believe, are either implementing the remaining fields (there may be a handful) and actually coming up with a list of all the fields to profile. Ideally I think we should calculate 1- and 2-D profiles for every field, and an HDF5 file with all of the profiles in it. The final output could then *additionally* include image plots of phase plots of mass-distribution with the average 1D profile overplotted. And perhaps projections. If we did all this, then we could not only aim to replicate functionality, but to enhance it. Best, Matt On Tue, Apr 12, 2011 at 1:50 PM, Britton Smith wrote:

Hi Rick, Cameron, everyone,

Cameron is correct.  I'm not suggesting any changes to Enzo.

Back before yt, enzo_anyl was used to make radial profiles of halos.  My understanding is that it only works in serial and that it is relatively difficult, or at least not straightforward, to add new fields for radial profiles.

The HaloProfiler tool in yt makes radial profiles in the same way, only it take user input to decide on what fields to profile.  It also can run in parallel and do fixed resolution projections around each halo.  Personally, I think this makes using the HaloProfiler a better choice moving forward. However, I have heard suggestions that the HaloProfiler lacks some functionality of enzo_anyl, preventing it from being a true replacement.  I don't think enzo_anyl should be gotten rid of if people are still using it. I just want to know what's missing so people who would like to use yt for their enzo_anyl needs don't feel like they are losing functionality by switching.

Britton

On Tue, Apr 12, 2011 at 1:36 PM, Cameron Hummels wrote:

...

Rick,

I don't think he's suggesting we change enzo.  I think he simply wants to know what functionality is lacking from yt's haloprofiler that enzoanyl currently has.  That said, I think when we get haloprofiler working up to spec with enzoanyl's capabilities, we should announce it to the enzo list, so people who use enzoanyl (there aren't many of us) will know that there is a better supported alternative in yt.

Cameron

On 04/12/2011 12:18 PM, Rick Wagner wrote:

...

Hi Britton,

On Apr 12, 2011, at 9:04 AM, Britton Smith wrote:

...

Hey everyone,

There is a fair amount of discussion about the HaloProfiler being a replacement for enzo_anyl.  I would definitely like to see this become official, since it seems to be keeping a number of people from making the switch to YT.  Can somebody outline the things that enzo_anyl does that the HaloProfiler cannot?  I have been under the impression that there wasn't anything, but I haven't used enzo_anyl in quite a while. I think that the HaloProfiler could easily be wrapped to look at act just like enzo_anyl, which makes this an attractive choice. But, I would post this question also the enzo-dev mailing list, since you're asking about changing Enzo, not YT.

--Rick

_______________________________________________ Yt-dev mailing list Yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org

-- Cameron Hummels PhD Candidate, Astronomy Department of Columbia University Public Outreach Director, Astronomy Department of Columbia University NASA IYA New York State Student Ambassador http://outreach.astro.columbia.edu PGP: 0x06F886E3

_______________________________________________ Yt-dev mailing list Yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org

_______________________________________________ Yt-dev mailing list Yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org

Hi Matt, Thanks for looking into this. I agree with your assessment of the cooling time issue, that the order should be 3, 2, 1 here. The number of ways that cooling can be done in Enzo alone is growing large enough that reconstructing all of that on our own in YT will be a major pain, and require updates every time someone modifies what is done in Enzo. That last part I definitely want to avoid. Option 2 is something that would be beneficial for many things, but is probably a little ways off. For now, I think it's fine to check whether the OutputCoolingTime parameter is set and profile that field based on that. When you you would like 2D profiles of every field, I'm not sure what you mean. By default, we're doing radial profiles of halos. Would we have default x and y fields for the 2D profiles? If so, I guess maybe density and temperature. Everything else you said I like a lot. Britton On Thu, Apr 14, 2011 at 11:45 AM, Matthew Turk wrote:

Hi all,

I had an opportunity to speak yesterday with Greg Bryan (who's not subscribed to this list) and independent of this thread, he brought up replacing enzo_anyl with yt. The one thing that I think we may be missing is the cooling time. To get the cooling time in yt we have three options:

1) Re-calculate from scratch, using all the items in the parameter file that govern the cooling time. This will also allow it to work for non-Enzo simulation outputs. 2) Wrap the Enzo cooling time routines. This will require utilizing fortran/C interop, possibly with f2py. It also has a lot of moving parts. (These would be solved if some day there was some agreement on microphysical solver APIs.) This would work with non-Enzo datasets. 3) Mandate that if you want the cooling time in your profiles, you run with OutputCoolingTime=1 in Enzo. This would not work with non-Enzo datasets, unless they too had an option to output the local cooling time in every cell.

I think #3 is the fastest time to solution, and certainly the option that is the least error prone. (The list above, it seems to me, is actually in order of increasing reliability.)

The remaining items that we would need to accomplish to make HaloProfiler a generalized replacement for enzo_anyl, I believe, are either implementing the remaining fields (there may be a handful) and actually coming up with a list of all the fields to profile. Ideally I think we should calculate 1- and 2-D profiles for every field, and an HDF5 file with all of the profiles in it. The final output could then *additionally* include image plots of phase plots of mass-distribution with the average 1D profile overplotted. And perhaps projections. If we did all this, then we could not only aim to replicate functionality, but to enhance it.

Best,

Matt

...

Hi Rick, Cameron, everyone,

Cameron is correct. I'm not suggesting any changes to Enzo.

Back before yt, enzo_anyl was used to make radial profiles of halos. My understanding is that it only works in serial and that it is relatively difficult, or at least not straightforward, to add new fields for radial profiles.

The HaloProfiler tool in yt makes radial profiles in the same way, only it take user input to decide on what fields to profile. It also can run in parallel and do fixed resolution projections around each halo. Personally, I think this makes using the HaloProfiler a better choice moving forward. However, I have heard suggestions that the HaloProfiler lacks some functionality of enzo_anyl, preventing it from being a true replacement. I don't think enzo_anyl should be gotten rid of if people are still using it. I just want to know what's missing so people who would like to use yt for their enzo_anyl needs don't feel like they are losing functionality by switching.

Britton

On Tue, Apr 12, 2011 at 1:36 PM, Cameron Hummels wrote:

...

Rick,

I don't think he's suggesting we change enzo. I think he simply wants to know what functionality is lacking from yt's haloprofiler that enzoanyl currently has. That said, I think when we get haloprofiler working up to spec with enzoanyl's capabilities, we should announce it to the enzo list, so people who use enzoanyl (there aren't many of us) will know that there is a better supported alternative in yt.

Cameron

On 04/12/2011 12:18 PM, Rick Wagner wrote:

...

Hi Britton,

On Apr 12, 2011, at 9:04 AM, Britton Smith wrote:

...

Hey everyone,

There is a fair amount of discussion about the HaloProfiler being a replacement for enzo_anyl. I would definitely like to see this

become

...

...

...

official, since it seems to be keeping a number of people from making

...

...

...

...

switch to YT. Can somebody outline the things that enzo_anyl does

On Tue, Apr 12, 2011 at 1:50 PM, Britton Smith wrote: the that the

...

...

...

...

HaloProfiler cannot? I have been under the impression that there wasn't anything, but I haven't used enzo_anyl in quite a while. I think that the HaloProfiler could easily be wrapped to look at act just like enzo_anyl, which makes this an attractive choice. But, I would post this question also the enzo-dev mailing list, since you're asking about changing Enzo, not YT.

--Rick

_______________________________________________ Yt-dev mailing list Yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org

-- Cameron Hummels PhD Candidate, Astronomy Department of Columbia University Public Outreach Director, Astronomy Department of Columbia University NASA IYA New York State Student Ambassador http://outreach.astro.columbia.edu PGP: 0x06F886E3

_______________________________________________ Yt-dev mailing list Yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org

_______________________________________________ Yt-dev mailing list Yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org

_______________________________________________ Yt-dev mailing list Yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org

Hi All,

1) Re-calculate from scratch, using all the items in the parameter file that govern the cooling time.  This will also allow it to work for non-Enzo simulation outputs. 2) Wrap the Enzo cooling time routines.  This will require utilizing fortran/C interop, possibly with f2py.  It also has a lot of moving parts.  (These would be solved if some day there was some agreement on microphysical solver APIs.)  This would work with non-Enzo datasets. 3) Mandate that if you want the cooling time in your profiles, you run with OutputCoolingTime=1 in Enzo.  This would not work with non-Enzo datasets, unless they too had an option to output the local cooling time in every cell.

I was going to reply that I thought we should choose between #3 or #1 (or do it in that order), but now I think #3 is a safer bet. What comes to mind is if you're going to want to post-calculate cooling times, you're going to have to make sure you set up your simulation correctly. Does your cooling depend on metallicity, and which metallicity fields? Gotta save those. So while you're at it, might as well just go ahead and save cooling times, too. The only reason not to save a field if you're otherwise able is for storage concerns, but if you're doing something that big, you'll be clever enough to figure out a solution to this conundrum on your own. And for non-Enzo codes, how hard is it to add a new output field? That's a real question, by the way. -- Stephen Skory s@skory.us http://stephenskory.com/ 510.621.3687 (google voice)

Hi John, On Thu, Apr 14, 2011 at 12:21 PM, John Wise wrote:

...

I was going to reply that I thought we should choose between #3 or #1 (or do it in that order), but now I think #3 is a safer bet. What comes to mind is if you're going to want to post-calculate cooling times, you're going to have to make sure you set up your simulation correctly. Does your cooling depend on metallicity, and which metallicity fields? Gotta save those. So while you're at it, might as well just go ahead and save cooling times, too. The only reason not to save a field if you're otherwise able is for storage concerns, but if you're doing something that big, you'll be clever enough to figure out a solution to this conundrum on your own.

I agree with this.  After thinking about it and reading Britton's response, I think #3 would be the best option.  A possible solution to calculating the cooling time post-runtime would be to add a command line option to Enzo that adds the CoolingTime to the data, similar to the potential field output command line option.  The catch is that you'd have to make sure that you use the same version as you ran the simulation.

I agree, that would be great. Do you think this is straightforward? I have looked before at the WritePotential stuff, but I'm not sure I'm the best person to add this. -Matt

John _______________________________________________ Yt-dev mailing list Yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org

Hi all, I've updated the list of fields: http://paste.enzotools.org/show/1583/ Some of these fields I don't know how to set up, but we either have most or could directly implement them. The tricky ones are the ones that separate out DM/Star particles, which will require a bit more touching of the data. (That's where most of the "NotImplemented" comes from.) Some of these fields we even had before -- like the inertial tensor -- and I think some people have a couple of these fields already in private repos. I pulled out all the disk analysis fields. It would probably be the work of an afternoon to wrap up replicating all the functionality and putting it into a "yt analyze" command. -Matt On Thu, Apr 14, 2011 at 2:54 PM, John Wise wrote:

...

...

I agree with this.  After thinking about it and reading Britton's response, I think #3 would be the best option.  A possible solution to calculating the cooling time post-runtime would be to add a command line option to Enzo that adds the CoolingTime to the data, similar to the potential field output command line option.  The catch is that you'd have to make sure that you use the same version as you ran the simulation.

I agree, that would be great.  Do you think this is straightforward? I have looked before at the WritePotential stuff, but I'm not sure I'm the best person to add this.

I could add this pretty quickly.  I'll try to do it today.

John _______________________________________________ Yt-dev mailing list Yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org

Hi Britton and John, John: Thanks very much for adding that to enzo! This will be hugely helpful. On Mon, Apr 18, 2011 at 3:52 PM, Britton Smith wrote:

Hi Matt,

Thanks for checking into this.  This looks like another thing that could be checked off our list with a code sprint.  I won't be available for that this week, but could do it next week.

Here are some questions that I think we need to consider.  While we are not only looking to provide full coverage for all of enzo_anyl's functionality so we can mothball it, we should also be thinking about things that could be improved from enzo_anyl.

1. The density units in enzo_anyl are Msun/Mpc^3.  Do we want to keep these?  I always converted to cgs back when I used enzo_anyl.

I prefer cgs, but that's because I'm biased. Someone else should answer this.

2. Stuff like clumping factors and velocity dispersion will require some additional functionality out of profiles.  It's currently not easy to get means and standard deviations in individual bins for profiles.  Does anyone have any idea how difficult it would be to implement this functionality?

Means I think are not too hard, but standard deviation is trickier. David and I talked about this a while back. I think to do these, we'll need to have two additional pieces of functionality: 1) "Derived" profile fields, for things like accretion rate. 2) Accumulator functions that get applied for every field. The actual binning occurs inside the profile objects, so we could add on accumulator functions. These will have to be fully-local, unfortunately. I'm not sure I am up for adding these at this point, but possibly sometime next month. The second is easier than the first. -Matt

I'm sure there are other things we should think about, but I don't have enough caffeine in my brain to think of them all.

Britton

On Mon, Apr 18, 2011 at 3:38 PM, Matthew Turk wrote:

...

Hi all,

I've updated the list of fields:

http://paste.enzotools.org/show/1583/

Some of these fields I don't know how to set up, but we either have most or could directly implement them.  The tricky ones are the ones that separate out DM/Star particles, which will require a bit more touching of the data.  (That's where most of the "NotImplemented" comes from.)  Some of these fields we even had before -- like the inertial tensor -- and I think some people have a couple of these fields already in private repos.  I pulled out all the disk analysis fields.

It would probably be the work of an afternoon to wrap up replicating all the functionality and putting it into a "yt analyze" command.

-Matt

On Thu, Apr 14, 2011 at 2:54 PM, John Wise wrote:

...

...

...

I agree with this.  After thinking about it and reading Britton's response, I think #3 would be the best option.  A possible solution to calculating the cooling time post-runtime would be to add a command line option to Enzo that adds the CoolingTime to the data, similar to the potential field output command line option.  The catch is that you'd have to make sure that you use the same version as you ran the simulation.

I agree, that would be great.  Do you think this is straightforward? I have looked before at the WritePotential stuff, but I'm not sure I'm the best person to add this.

I could add this pretty quickly.  I'll try to do it today.

John _______________________________________________ Yt-dev mailing list Yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org

_______________________________________________ Yt-dev mailing list Yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org

_______________________________________________ Yt-dev mailing list Yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org

Thanks everyone for working on this functionality. On Apr 19, 2011, at 2:49 PM, Matthew Turk wrote:

Hi Britton and John,

John: Thanks very much for adding that to enzo! This will be hugely helpful.

On Mon, Apr 18, 2011 at 3:52 PM, Britton Smith wrote:

...

Hi Matt,

Thanks for checking into this. This looks like another thing that could be checked off our list with a code sprint. I won't be available for that this week, but could do it next week.

Here are some questions that I think we need to consider. While we are not only looking to provide full coverage for all of enzo_anyl's functionality so we can mothball it, we should also be thinking about things that could be improved from enzo_anyl.

1. The density units in enzo_anyl are Msun/Mpc^3. Do we want to keep these? I always converted to cgs back when I used enzo_anyl.

I prefer cgs, but that's because I'm biased. Someone else should answer this.

I think cgs is good (actually SI would be better -- I think astronomy is headed toward convergence with the rest of the scientific world, and we will, in time, also adopt SI).

...

2. Stuff like clumping factors and velocity dispersion will require some additional functionality out of profiles. It's currently not easy to get means and standard deviations in individual bins for profiles. Does anyone have any idea how difficult it would be to implement this functionality?

Means I think are not too hard, but standard deviation is trickier. David and I talked about this a while back.

I would think standard deviation is no more difficult than mean. It can be computed in one pass since <(x-<x>)^2> = - <x>^2. Although I may just be missing the issue. Greg

Hi Stephen, On Thu, Apr 14, 2011 at 12:01 PM, Stephen Skory wrote:

Hi All,

...

1) Re-calculate from scratch, using all the items in the parameter file that govern the cooling time.  This will also allow it to work for non-Enzo simulation outputs. 2) Wrap the Enzo cooling time routines.  This will require utilizing fortran/C interop, possibly with f2py.  It also has a lot of moving parts.  (These would be solved if some day there was some agreement on microphysical solver APIs.)  This would work with non-Enzo datasets. 3) Mandate that if you want the cooling time in your profiles, you run with OutputCoolingTime=1 in Enzo.  This would not work with non-Enzo datasets, unless they too had an option to output the local cooling time in every cell.

I was going to reply that I thought we should choose between #3 or #1 (or do it in that order), but now I think #3 is a safer bet. What comes to mind is if you're going to want to post-calculate cooling times, you're going to have to make sure you set up your simulation correctly. Does your cooling depend on metallicity, and which metallicity fields? Gotta save those. So while you're at it, might as well just go ahead and save cooling times, too. The only reason not to save a field if you're otherwise able is for storage concerns, but if you're doing something that big, you'll be clever enough to figure out a solution to this conundrum on your own.

I personally would say that a combination of #1 and #2, where we have a regular API for microphysical solvers would be the ideal solution. I think that, as John mentioned in IRC, it would also be nice to be able to separate out individual lines and reactions to get sub-model cooling info, too.

And for non-Enzo codes, how hard is it to add a new output field? That's a real question, by the way.

I think it varies. My recollection is that Orion/CASTRO/Maestro is that it's pretty straightforward, but that FLASH has some challenges. For ART and RAMSES I don't know. I suspect it's more difficult for those two codes, because my understanding is the the IO is very tightly coupled to the internal data structures. -Matt

-- Stephen Skory s@skory.us http://stephenskory.com/ 510.621.3687 (google voice) _______________________________________________ Yt-dev mailing list Yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org