Hi Geoffrey, This is nearly what I'm looking for, but not quite. I have halos that were generated outside of yt, so they are not in the format. Is there either a simpler way of loading them in and getting halo objects, or a standard file format that I could subscribe to? Better yet, do we have a way of yielding halo data to get halos back? i.e., something like numpy's "fromiter" function? Thanks! -Matt On Mon, Jul 9, 2012 at 2:20 AM, Geoffrey So <gsiisg@gmail.com> wrote:

I thought (correct me if I'm wrong Stephen), that if you didn't save the halo objects with the call in the halo finder halo_list.dump(HaloListname) They are lost and not saved, so to save the haloes for access later in .h5 files, that previous .dump() command must be called. If they were saved (check for .h5 files), you should be able to load the haloes with haloes = LoadHaloes(pf, HaloListname)

From G.S.

On Sun, Jul 8, 2012 at 6:28 PM, Matthew Turk <matthewturk@gmail.com> wrote:

...

Hi all,

If I have a list of halo locations and radii, what is the best way to load that into yt? Specifically, to get back a set of halo objects?

Thanks,

Matt _______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org

_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org

Hi Matt, I think I understand what you're asking. You would like a function that looks something like halos = LoadPlainHaloes(filename, {'x':0, 'y':1, 'z':2, 'r':3}) where the dict specifies which column for which data field? That indeed does not exist, but I think it shouldn't be too hard to add, considering it's (basically) a simplification of LoadHaloes that already exists. Would you like me to take a whack at it, or have you already got a working solution? -- Stephen Skory s@skory.us http://stephenskory.com/ 510.621.3687 (google voice)

Hi Matt,

I would love some help with it, but I was going to try to do it tonight. If you happen to have a few spare cycles, that would be awesome, but I understand if not!

I'll see what I can whip up. -- Stephen Skory s@skory.us http://stephenskory.com/ 510.621.3687 (google voice)

Hi Stephen, I read the code and it looks awesome. Thanks very much -- I'm +1 on including this in main yt. -Matt On Mon, Jul 9, 2012 at 3:09 PM, Stephen Skory <s@skory.us> wrote:

Hi Matt,

I've pushed a change here:

https://bitbucket.org/sskory/yt/changeset/98e4616f576d

it's basically how I described it in my earlier email.

pf = load("data0005") halos = LoadHaloes(pf, "list.txt", {'x':0, 'y':1, 'z':2, 'r':3}, comment = ";")

those are the only four types of columns it knows how to read right now. That can be changed if needed. "comment" lists lines to ignore by leading character and defaults to "#". In my limited testing all the halo operations work. If you find a bug, let me know! If you like it, I'll issue a PR.

-- Stephen Skory s@skory.us http://stephenskory.com/ 510.621.3687 (google voice) _______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org

A suggestion, if it isn't too much trouble. Could you add a column for mass? Christine On Mon, 2012-07-09 at 16:11 -0600, Stephen Skory wrote:

Ooops, try this:

halos = LoadTextHaloes(pf, "list.txt", {'x':0, 'y':1, 'z':2, 'r':3}, comment = ";")

Christine,

A suggestion, if it isn't too much trouble. Could you add a column for mass?

Actually, just as I sent my last reply I thought a bit harder, and I realized that this raises a prickly issue. Let me explain. When you run a halo finder in yt, and save your results with .dump(), you can use LoadHaloes() to recover carbon-copies of what you had before. Any operation you ran on the halos before .dump() will give you the same answer with them loaded later on. This is because with .dump() everything that makes up the halos, in particular the full particle assignments, is saved to disk. However, with this text loading method, we don't know anything about what particles are in the halo when the halo finder was run. This means that all we can do (and it's what I'm doing in this new method) is simply define the halo as a sphere centered on a point. If I ask for particle information from this halo, it will use the sphere. Typically, halos found by halo finders are not spheres. Now, if we read in mass from a text file as you suggest, what mass is this? If it's the mass of the particles as the halo finder you used calculated, what does that mean in yt? It can't reproduce this mass on its own, and this mass doesn't correspond to anything yt knows about, namely, a nondescript sphere. I'll have to think about this some more, but the TL;DR is that specifying a mass like this from a text file introduces some ambiguity. I'm still open to other column suggestions. -- Stephen Skory s@skory.us http://stephenskory.com/ 510.621.3687 (google voice)

Hi everyone, The radius, even if somewhat arbitrary is a very useful quantity to have on hand for a halo. With the halo profiler, for example, the halo radius is used to determine the maximum radius out to which radial profiling should be done. Without that, there's no good way to decide how far out radial profiles should be taking. Setting a constant value is generally not a good idea, since it can vary a lot from situation to situation. With the halo profiler, you can then calculate virial mass and radius in a way that also includes the baryons, something that the halo finder cannot do. In conclusion, I strongly support reading in a radius from a halo catalog. Britton On Mon, Jul 9, 2012 at 7:06 PM, Christine Simpson < csimpson@astro.columbia.edu> wrote:

Hi Stephen,

I didn't mean to butt into your discussion with Matt, sorry. I don't want to make more work for you and I totally understand if this suggestion doesn't fit in with your vision of the yt halo routines.

I will say, however, that I actually have somewhat of the opposite opinion on the issue that you raise. To me the radius is an arbitrarily calculated value and the mass that the halo finder returns is what is more fundamental. Of course, yt can't know the particles that an external halo finder identified as belonging to a halo unless that information is provided, but I would argue that having a 'radius' doesn't give you that information either.

It's been a while since I've looked into the halo finding and handling routines in yt, so it is very likely that my information is outdated (I apologize if this is the case), but I recall that the default for the halo 'radius' was the radius of the furthest group particle. For a halo finder like hop that can return very irregular halos, this makes little physical sense in my opinion. I believe there was at one time an option to use what the docs called the 'r200' radius, but it was defined relative to the mean density of the universe. I guess it is okay to define it this way, but it is rather non-standard (I would define it relative to the critical density of the universe). Either way, this radius is defined by the mass of the halo (and the redshift). I understand that cosmology is messy, but I think it is better to use that radius than the distance of the furthest particle, which could be quite arbitrary in many respects.

I guess I would just say that for me, I'd rather know what total mass the halo finder identified and then work from there (i.e. define a radius and then a sphere object). I can see that others may have other opinions and halo finders that may operate in other ways. I guess my one piece of advice would be to try to make routines as general as possible. I sometimes feel in yt that there is an unnecessary degree of specialization. Of course, I love yt and I say this from a place of sincere appreciation for the package and its developers.

Christine

On Mon, 2012-07-09 at 16:23 -0600, Stephen Skory wrote:

...

Christine,

...

A suggestion, if it isn't too much trouble. Could you add a column for mass?

Actually, just as I sent my last reply I thought a bit harder, and I realized that this raises a prickly issue. Let me explain.

When you run a halo finder in yt, and save your results with .dump(), you can use LoadHaloes() to recover carbon-copies of what you had before. Any operation you ran on the halos before .dump() will give you the same answer with them loaded later on. This is because with .dump() everything that makes up the halos, in particular the full particle assignments, is saved to disk.

However, with this text loading method, we don't know anything about what particles are in the halo when the halo finder was run. This means that all we can do (and it's what I'm doing in this new method) is simply define the halo as a sphere centered on a point. If I ask for particle information from this halo, it will use the sphere. Typically, halos found by halo finders are not spheres.

Now, if we read in mass from a text file as you suggest, what mass is this? If it's the mass of the particles as the halo finder you used calculated, what does that mean in yt? It can't reproduce this mass on its own, and this mass doesn't correspond to anything yt knows about, namely, a nondescript sphere.

I'll have to think about this some more, but the TL;DR is that specifying a mass like this from a text file introduces some ambiguity.

I'm still open to other column suggestions.

_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org

Hi all, tl;dr: Mass column sounds fine, as long as we recognize that it's not going to agree with what yt would calculate from the other info. Christine, you're absolutely not "butting in" -- thank you for your thoughts. Initially when I saw your exchange with Stephen, my thought was that the radius is the more fundamental component when dealing with halos, since one could in principle extract back out the mass. This operates under the assumption -- which you note -- that the radii of the particle that's furthest from the center governs the halos. As Geoffrey pointed out, this is not a terribly good assumption. That being said, I still consider the radius to be more fundamental, from the perspective of reconstructing a useful halo object. For the purposes of visualization, it's fine. All of our viz that uses baryon quantities will not be bothered by this. But for things that look at baryonic quantities, we might not really want the halo (which is triaxially) to be defined only by a single quantity. So, reconstructing a halo from a radius is probably sufficient for some tasks, but it is lossy, and poor. Really, though, the question we're dancing around here is somewhat more intrusive. The point of finding halos and loading them into yt is not so that we can *have* halos in yt, it's so that we can do something with them. That something could be making visualizations, tracking particles, analyzing baryonic quantities and so on. Some of these need the exact particles that belonged to a halo, others don't. For instance, right now identify which fluid cells/parcels "belong" to a given halo is somewhat ill-defined. We use the radius to figure out which cells belong to the halo, and we use spheres. Hopefully in the future we can use Geoffrey's ellipsoidal stuff, but that's not fully in place yet. So when we do any kind of analysis, now, on fluid that's affiliated with particle collections, we have to make assumptions about which fluid parcels contribute. Britton's points are also quite good, in that one can provide a secondary calibration of the halo size based on virial arguments. So perhaps it's not necessarily interesting to talk about loading in halos unless they explicitly come with lists of particles, and even then saying which cells (in a mesh code) belong to a given halo is not necessarily completely well-defined. For my purposes, I think I'm okay with loading in spheres as "halo" objects, getting the particles from the file, and then applying traditional "halo" analysis to that. But, we're kind of at a funny spot here. (And maybe this should be moved over to yt-dev.) I think there is a place for loading in some kind of information from an external source, and using that to guide subsequent analysis. In this case, the source was a halo finder, and I wanted to apply analysis of things like examining ionization fraction and SFR. In fact, I think I *prefer* that to the idea that yt needs to have every possible halo finder, every possible [thing] in it, and that all analysis has to happen internally. That kind of philosophy doesn't scale, and isn't terribly inclusive. And, really, I think we open up a lot of really interesting collaborations between yt and (for instance) SAM codes if we start thinking about halo objects not necessarily as collections of particles, but also semantically meaningful objects. Something to think about when discussing halo finding in yt, I suppose. Well, anyway, this has kind of diverged. I'm okay with having a mass column, but I want to point out that the mass column won't agree with the mass calculated by yt; the only way for that to happen is to have the halo catalog also contain particle information. And the only way to take particle information and get back those specific particles is to do search through particle IDs to match them up. This should be fast, but acceptable. Anyway, thanks to everyone for your thoughts -- and to Stephen for the implementation! -Matt On Mon, Jul 9, 2012 at 8:07 PM, Geoffrey So <gsiisg@gmail.com> wrote:

Getting back to "getting the halo", I have a dumb thought. I was thinking since the maximum radius is known, YT can extract the particles/fields in the sphere and call that a new halo, but you can also do a halo finding in that sphere and just take the biggest halo as the halo that was meant by the center and max radius right? But this of course requires you to know how the original haloes were generated (specifically which halo finder it used), and if an matching method exists in YT.... Come to think of it, if you know the original halo finding method and it exists in YT, you can just run the halo finder in YT and use .dump() to save the info, the only reason not to do that is to save computation time, but those are cheap nowadays.

From G.S.

On Mon, Jul 9, 2012 at 5:52 PM, Britton Smith <brittonsmith@gmail.com> wrote:

...

Hi everyone,

The radius, even if somewhat arbitrary is a very useful quantity to have on hand for a halo. With the halo profiler, for example, the halo radius is used to determine the maximum radius out to which radial profiling should be done. Without that, there's no good way to decide how far out radial profiles should be taking. Setting a constant value is generally not a good idea, since it can vary a lot from situation to situation. With the halo profiler, you can then calculate virial mass and radius in a way that also includes the baryons, something that the halo finder cannot do. In conclusion, I strongly support reading in a radius from a halo catalog.

Britton

On Mon, Jul 9, 2012 at 7:06 PM, Christine Simpson <csimpson@astro.columbia.edu> wrote:

...

Hi Stephen,

I didn't mean to butt into your discussion with Matt, sorry. I don't want to make more work for you and I totally understand if this suggestion doesn't fit in with your vision of the yt halo routines.

I will say, however, that I actually have somewhat of the opposite opinion on the issue that you raise. To me the radius is an arbitrarily calculated value and the mass that the halo finder returns is what is more fundamental. Of course, yt can't know the particles that an external halo finder identified as belonging to a halo unless that information is provided, but I would argue that having a 'radius' doesn't give you that information either.

It's been a while since I've looked into the halo finding and handling routines in yt, so it is very likely that my information is outdated (I apologize if this is the case), but I recall that the default for the halo 'radius' was the radius of the furthest group particle. For a halo finder like hop that can return very irregular halos, this makes little physical sense in my opinion. I believe there was at one time an option to use what the docs called the 'r200' radius, but it was defined relative to the mean density of the universe. I guess it is okay to define it this way, but it is rather non-standard (I would define it relative to the critical density of the universe). Either way, this radius is defined by the mass of the halo (and the redshift). I understand that cosmology is messy, but I think it is better to use that radius than the distance of the furthest particle, which could be quite arbitrary in many respects.

I guess I would just say that for me, I'd rather know what total mass the halo finder identified and then work from there (i.e. define a radius and then a sphere object). I can see that others may have other opinions and halo finders that may operate in other ways. I guess my one piece of advice would be to try to make routines as general as possible. I sometimes feel in yt that there is an unnecessary degree of specialization. Of course, I love yt and I say this from a place of sincere appreciation for the package and its developers.

Christine

On Mon, 2012-07-09 at 16:23 -0600, Stephen Skory wrote:

...

Christine,

...

A suggestion, if it isn't too much trouble. Could you add a column for mass?

Actually, just as I sent my last reply I thought a bit harder, and I realized that this raises a prickly issue. Let me explain.

When you run a halo finder in yt, and save your results with .dump(), you can use LoadHaloes() to recover carbon-copies of what you had before. Any operation you ran on the halos before .dump() will give you the same answer with them loaded later on. This is because with .dump() everything that makes up the halos, in particular the full particle assignments, is saved to disk.

However, with this text loading method, we don't know anything about what particles are in the halo when the halo finder was run. This means that all we can do (and it's what I'm doing in this new method) is simply define the halo as a sphere centered on a point. If I ask for particle information from this halo, it will use the sphere. Typically, halos found by halo finders are not spheres.

Now, if we read in mass from a text file as you suggest, what mass is this? If it's the mass of the particles as the halo finder you used calculated, what does that mean in yt? It can't reproduce this mass on its own, and this mass doesn't correspond to anything yt knows about, namely, a nondescript sphere.

I'll have to think about this some more, but the TL;DR is that specifying a mass like this from a text file introduces some ambiguity.

I'm still open to other column suggestions.

_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org

_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org

_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org