Heya everyone! I am trying to track clumps across multiple snapshots (from flash). Now my questions is, has anybody done something like this and would be willing to share his/her code? Best Wishes, Patrick
Hi Patrick, Thanks for writing, and welcome to yt-users. :) On Thu, Oct 4, 2012 at 9:42 AM, Patrick Rieser <patrick.rieser@uibk.ac.at> wrote:
Heya everyone!
I am trying to track clumps across multiple snapshots (from flash). Now my questions is, has anybody done something like this and would be willing to share his/her code?
David Collins wrote some code that did this, a couple years ago. I don't know the current status. If you wanted to write a new set of code to do this (and you didn't want to go full-on "lagrangian coherent structures") there are some things in yt that could help out. For instance, we include a Cython kD-tree that we use to provide a nearest-neighbor search when doing merger trees. This is used in a very simple way in the code in yt/analysis_modules/halo_merger_tree/enzofof_merger_tree.py , where halos are loaded into a variable called halo_kdtree. This then gets searched with a ball query. You could in principle load the clumps into the same kdtree structure, perform the search, and then apply selection criteria for clump tracking based on that. (Of course this is just the first step in identifying clump motion -- but it would be a way to reduce from N^2 searching.) Let us know if you run into any tricks or have any successes -- this is a pretty cool idea, and I'd love to see where it leads you! -Matt
Best Wishes, Patrick _______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
As I am running a little bit out of time to finish this, I thought about doint this the simple and "brute force" way. So this is the current plan: I am going to iterate through all snapshots we have, and write out all clump objects with pickle. Starting at snapshot 0 i am going to estimate the position of the center of mass of the clump in the new snapshot using it's bulk velocity and see if I can find the clump in the surrounding area (looping through all clumps in the following snapshot). If I find one or more, I will compare the mass and identify it as the same or not (if the mass doesn't match maybe i check if a second one vanished). This might not be a very elegant way, but I hope it works for our system (galaxy cluster). I am quite new to this stuff, so if I got any horrible mistakes here, please correct me.
For instance, we include a Cython kD-tree that we use to provide a nearest-neighbor search when doing merger trees.
Thanks, I will take a look into that!
and you didn't want to go full-on "lagrangian coherent structures"
I took a look at it and it seems really interesting. It's really a pity I don't have enough time at the moment. best wishes, Patrick On 2012-10-05 18:07, Matthew Turk wrote:
Hi Patrick,
Thanks for writing, and welcome to yt-users. :)
On Thu, Oct 4, 2012 at 9:42 AM, Patrick Rieser <patrick.rieser@uibk.ac.at> wrote:
Heya everyone!
I am trying to track clumps across multiple snapshots (from flash). Now my questions is, has anybody done something like this and would be willing to share his/her code? David Collins wrote some code that did this, a couple years ago. I don't know the current status.
If you wanted to write a new set of code to do this (and you didn't want to go full-on "lagrangian coherent structures") there are some things in yt that could help out. For instance, we include a Cython kD-tree that we use to provide a nearest-neighbor search when doing merger trees. This is used in a very simple way in the code in yt/analysis_modules/halo_merger_tree/enzofof_merger_tree.py , where halos are loaded into a variable called halo_kdtree. This then gets searched with a ball query. You could in principle load the clumps into the same kdtree structure, perform the search, and then apply selection criteria for clump tracking based on that. (Of course this is just the first step in identifying clump motion -- but it would be a way to reduce from N^2 searching.)
Let us know if you run into any tricks or have any successes -- this is a pretty cool idea, and I'd love to see where it leads you!
-Matt
Best Wishes, Patrick _______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
Hi Patrick, I did something very similar when I tried to track clumps (molecular clump in star formation simulation in my case) using the centre of mass of the clump (we ended up using the position of the densest pixel instead of the centre of mass since CoM tracking gave a huge fluctuation in mass) and using the velocity vector of that centre to infer the direction of the progenitor of that clump. What I noticed was that the mass is still quite unstable due to the intrinsic definition of the clump. I found that starting lv 3, the mass fluctuation is so great I couldn't trust my tracking any more. I'm not sure if you'll encounter the same problem. David Collins has been working on this extensively using tracer particles so he should be able to help you more. Best Regards, Eve On 10/5/2012 4:15 PM, Patrick Rieser wrote:
As I am running a little bit out of time to finish this, I thought about doint this the simple and "brute force" way. So this is the current plan: I am going to iterate through all snapshots we have, and write out all clump objects with pickle. Starting at snapshot 0 i am going to estimate the position of the center of mass of the clump in the new snapshot using it's bulk velocity and see if I can find the clump in the surrounding area (looping through all clumps in the following snapshot). If I find one or more, I will compare the mass and identify it as the same or not (if the mass doesn't match maybe i check if a second one vanished). This might not be a very elegant way, but I hope it works for our system (galaxy cluster). I am quite new to this stuff, so if I got any horrible mistakes here, please correct me.
For instance, we include a Cython kD-tree that we use to provide a nearest-neighbor search when doing merger trees.
Thanks, I will take a look into that!
and you didn't want to go full-on "lagrangian coherent structures"
I took a look at it and it seems really interesting. It's really a pity I don't have enough time at the moment.
best wishes, Patrick
On 2012-10-05 18:07, Matthew Turk wrote:
Hi Patrick,
Thanks for writing, and welcome to yt-users. :)
On Thu, Oct 4, 2012 at 9:42 AM, Patrick Rieser <patrick.rieser@uibk.ac.at> wrote:
Heya everyone!
I am trying to track clumps across multiple snapshots (from flash). Now my questions is, has anybody done something like this and would be willing to share his/her code? David Collins wrote some code that did this, a couple years ago. I don't know the current status.
If you wanted to write a new set of code to do this (and you didn't want to go full-on "lagrangian coherent structures") there are some things in yt that could help out. For instance, we include a Cython kD-tree that we use to provide a nearest-neighbor search when doing merger trees. This is used in a very simple way in the code in yt/analysis_modules/halo_merger_tree/enzofof_merger_tree.py , where halos are loaded into a variable called halo_kdtree. This then gets searched with a ball query. You could in principle load the clumps into the same kdtree structure, perform the search, and then apply selection criteria for clump tracking based on that. (Of course this is just the first step in identifying clump motion -- but it would be a way to reduce from N^2 searching.)
Let us know if you run into any tricks or have any successes -- this is a pretty cool idea, and I'd love to see where it leads you!
-Matt
Best Wishes, Patrick _______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
Yeah, clump tracking is one of the logs in my project log jam right now. As Eve said, I also had problems with mass stability because of the definition of "clump." Elizabeth Tasker has had success with tracking clumps in her disk galaxy simulations--she does a single density threshold for the clump definition and center of mass trajectory to track clumps, and those results seem pretty solid. So it may work for some systems. Her flow is relatively laminar, especially at the beginning, which I believe makes the problem much easier. I imagine you will have a similar problem with significant fluctuations in galaxy clusters, though the turbulence is of a lower mach number than I was dealing with, so it might not be so bad. I initially did something basically as you described-- pickle the clumps to disk, simple n^2 loop, comparing time-advanced center of mass from clumps in t=1 to extents in t=2. I would recommend against eliminating based on mass since you don't have a surface that's actually legrangian. For my purposes, n^2 was fast enough since I only had a few hundred timsteps and a few hundred clumps in each snapshot. Ran in an hour or two. I can send you my code, but I'm quite sure the effort to get my overly complicated 3 year old thing running again is higher than doing it fresh. It is true that I've been looking at using tracer particles. I can't tell if it works yet. It's either super cool or totally useless. d. On Sat, Oct 6, 2012 at 2:30 AM, Eve Lee <evelee@berkeley.edu> wrote:
Hi Patrick,
I did something very similar when I tried to track clumps (molecular clump in star formation simulation in my case) using the centre of mass of the clump (we ended up using the position of the densest pixel instead of the centre of mass since CoM tracking gave a huge fluctuation in mass) and using the velocity vector of that centre to infer the direction of the progenitor of that clump.
What I noticed was that the mass is still quite unstable due to the intrinsic definition of the clump. I found that starting lv 3, the mass fluctuation is so great I couldn't trust my tracking any more. I'm not sure if you'll encounter the same problem.
David Collins has been working on this extensively using tracer particles so he should be able to help you more.
Best Regards,
Eve
On 10/5/2012 4:15 PM, Patrick Rieser wrote:
As I am running a little bit out of time to finish this, I thought about doint this the simple and "brute force" way. So this is the current plan: I am going to iterate through all snapshots we have, and write out all clump objects with pickle. Starting at snapshot 0 i am going to estimate the position of the center of mass of the clump in the new snapshot using it's bulk velocity and see if I can find the clump in the surrounding area (looping through all clumps in the following snapshot). If I find one or more, I will compare the mass and identify it as the same or not (if the mass doesn't match maybe i check if a second one vanished). This might not be a very elegant way, but I hope it works for our system (galaxy cluster). I am quite new to this stuff, so if I got any horrible mistakes here, please correct me.
For instance, we include a Cython kD-tree that we use to provide a nearest-neighbor search when doing merger trees.
Thanks, I will take a look into that!
and you didn't want to go full-on "lagrangian coherent structures"
I took a look at it and it seems really interesting. It's really a pity I don't have enough time at the moment.
best wishes, Patrick
On 2012-10-05 18:07, Matthew Turk wrote:
Hi Patrick,
Thanks for writing, and welcome to yt-users. :)
On Thu, Oct 4, 2012 at 9:42 AM, Patrick Rieser <patrick.rieser@uibk.ac.at> wrote:
Heya everyone!
I am trying to track clumps across multiple snapshots (from flash). Now my questions is, has anybody done something like this and would be willing to share his/her code?
David Collins wrote some code that did this, a couple years ago. I don't know the current status.
If you wanted to write a new set of code to do this (and you didn't want to go full-on "lagrangian coherent structures") there are some things in yt that could help out. For instance, we include a Cython kD-tree that we use to provide a nearest-neighbor search when doing merger trees. This is used in a very simple way in the code in yt/analysis_modules/halo_merger_tree/enzofof_merger_tree.py , where halos are loaded into a variable called halo_kdtree. This then gets searched with a ball query. You could in principle load the clumps into the same kdtree structure, perform the search, and then apply selection criteria for clump tracking based on that. (Of course this is just the first step in identifying clump motion -- but it would be a way to reduce from N^2 searching.)
Let us know if you run into any tricks or have any successes -- this is a pretty cool idea, and I'd love to see where it leads you!
-Matt
Best Wishes, Patrick _______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
-- Sent from my computer.
It would be awesome if you could send me your code! I am sure it would be of quite some help to take at least a look at it. And yes, I really hope that I won't run into a lot of problems with my systems. Even though I got some thousand snapshots, I have access to nice server, where i can grab at least 64 cores where I can track the clumps from different clusters simutanously, so this is hopefully no problem. When you talk about tracer particles, you mean that you put an "imaginary" particle in e.g. every cell and let it follow the velocity field to determine where the cells that make up a clump end up? Best regards, Patrick Am 06.10.2012 19:11, schrieb David Collins:
Yeah, clump tracking is one of the logs in my project log jam right now. As Eve said, I also had problems with mass stability because of the definition of "clump." Elizabeth Tasker has had success with tracking clumps in her disk galaxy simulations--she does a single density threshold for the clump definition and center of mass trajectory to track clumps, and those results seem pretty solid. So it may work for some systems. Her flow is relatively laminar, especially at the beginning, which I believe makes the problem much easier. I imagine you will have a similar problem with significant fluctuations in galaxy clusters, though the turbulence is of a lower mach number than I was dealing with, so it might not be so bad.
I initially did something basically as you described-- pickle the clumps to disk, simple n^2 loop, comparing time-advanced center of mass from clumps in t=1 to extents in t=2. I would recommend against eliminating based on mass since you don't have a surface that's actually legrangian. For my purposes, n^2 was fast enough since I only had a few hundred timsteps and a few hundred clumps in each snapshot. Ran in an hour or two. I can send you my code, but I'm quite sure the effort to get my overly complicated 3 year old thing running again is higher than doing it fresh.
It is true that I've been looking at using tracer particles. I can't tell if it works yet. It's either super cool or totally useless.
d.
On Sat, Oct 6, 2012 at 2:30 AM, Eve Lee <evelee@berkeley.edu> wrote:
Hi Patrick,
I did something very similar when I tried to track clumps (molecular clump in star formation simulation in my case) using the centre of mass of the clump (we ended up using the position of the densest pixel instead of the centre of mass since CoM tracking gave a huge fluctuation in mass) and using the velocity vector of that centre to infer the direction of the progenitor of that clump.
What I noticed was that the mass is still quite unstable due to the intrinsic definition of the clump. I found that starting lv 3, the mass fluctuation is so great I couldn't trust my tracking any more. I'm not sure if you'll encounter the same problem.
David Collins has been working on this extensively using tracer particles so he should be able to help you more.
Best Regards,
Eve
On 10/5/2012 4:15 PM, Patrick Rieser wrote:
As I am running a little bit out of time to finish this, I thought about doint this the simple and "brute force" way. So this is the current plan: I am going to iterate through all snapshots we have, and write out all clump objects with pickle. Starting at snapshot 0 i am going to estimate the position of the center of mass of the clump in the new snapshot using it's bulk velocity and see if I can find the clump in the surrounding area (looping through all clumps in the following snapshot). If I find one or more, I will compare the mass and identify it as the same or not (if the mass doesn't match maybe i check if a second one vanished). This might not be a very elegant way, but I hope it works for our system (galaxy cluster). I am quite new to this stuff, so if I got any horrible mistakes here, please correct me.
For instance, we include a Cython kD-tree that we use to provide a nearest-neighbor search when doing merger trees.
Thanks, I will take a look into that!
and you didn't want to go full-on "lagrangian coherent structures"
I took a look at it and it seems really interesting. It's really a pity I don't have enough time at the moment.
best wishes, Patrick
On 2012-10-05 18:07, Matthew Turk wrote:
Hi Patrick,
Thanks for writing, and welcome to yt-users. :)
On Thu, Oct 4, 2012 at 9:42 AM, Patrick Rieser <patrick.rieser@uibk.ac.at> wrote:
Heya everyone!
I am trying to track clumps across multiple snapshots (from flash). Now my questions is, has anybody done something like this and would be willing to share his/her code? David Collins wrote some code that did this, a couple years ago. I don't know the current status.
If you wanted to write a new set of code to do this (and you didn't want to go full-on "lagrangian coherent structures") there are some things in yt that could help out. For instance, we include a Cython kD-tree that we use to provide a nearest-neighbor search when doing merger trees. This is used in a very simple way in the code in yt/analysis_modules/halo_merger_tree/enzofof_merger_tree.py , where halos are loaded into a variable called halo_kdtree. This then gets searched with a ball query. You could in principle load the clumps into the same kdtree structure, perform the search, and then apply selection criteria for clump tracking based on that. (Of course this is just the first step in identifying clump motion -- but it would be a way to reduce from N^2 searching.)
Let us know if you run into any tricks or have any successes -- this is a pretty cool idea, and I'd love to see where it leads you!
-Matt
Best Wishes, Patrick _______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
I'll send it off list. There are a bunch of things in it that don't work and need some explanation, and I don't want to pollute the list. Anyone else who wants this thing, chime in.
When you talk about tracer particles, you mean that you put an "imaginary"
That's exactly right. So they need to be in before you start the simulations. d.
particle in e.g. every cell and let it follow the velocity field to determine where the cells that make up a clump end up?
Best regards, Patrick
Am 06.10.2012 19:11, schrieb David Collins:
Yeah, clump tracking is one of the logs in my project log jam right now. As Eve said, I also had problems with mass stability because of the definition of "clump." Elizabeth Tasker has had success with tracking clumps in her disk galaxy simulations--she does a single density threshold for the clump definition and center of mass trajectory to track clumps, and those results seem pretty solid. So it may work for some systems. Her flow is relatively laminar, especially at the beginning, which I believe makes the problem much easier. I imagine you will have a similar problem with significant fluctuations in galaxy clusters, though the turbulence is of a lower mach number than I was dealing with, so it might not be so bad.
I initially did something basically as you described-- pickle the clumps to disk, simple n^2 loop, comparing time-advanced center of mass from clumps in t=1 to extents in t=2. I would recommend against eliminating based on mass since you don't have a surface that's actually legrangian. For my purposes, n^2 was fast enough since I only had a few hundred timsteps and a few hundred clumps in each snapshot. Ran in an hour or two. I can send you my code, but I'm quite sure the effort to get my overly complicated 3 year old thing running again is higher than doing it fresh.
It is true that I've been looking at using tracer particles. I can't tell if it works yet. It's either super cool or totally useless.
d.
On Sat, Oct 6, 2012 at 2:30 AM, Eve Lee <evelee@berkeley.edu> wrote:
Hi Patrick,
I did something very similar when I tried to track clumps (molecular clump in star formation simulation in my case) using the centre of mass of the clump (we ended up using the position of the densest pixel instead of the centre of mass since CoM tracking gave a huge fluctuation in mass) and using the velocity vector of that centre to infer the direction of the progenitor of that clump.
What I noticed was that the mass is still quite unstable due to the intrinsic definition of the clump. I found that starting lv 3, the mass fluctuation is so great I couldn't trust my tracking any more. I'm not sure if you'll encounter the same problem.
David Collins has been working on this extensively using tracer particles so he should be able to help you more.
Best Regards,
Eve
On 10/5/2012 4:15 PM, Patrick Rieser wrote:
As I am running a little bit out of time to finish this, I thought about doint this the simple and "brute force" way. So this is the current plan: I am going to iterate through all snapshots we have, and write out all clump objects with pickle. Starting at snapshot 0 i am going to estimate the position of the center of mass of the clump in the new snapshot using it's bulk velocity and see if I can find the clump in the surrounding area (looping through all clumps in the following snapshot). If I find one or more, I will compare the mass and identify it as the same or not (if the mass doesn't match maybe i check if a second one vanished). This might not be a very elegant way, but I hope it works for our system (galaxy cluster). I am quite new to this stuff, so if I got any horrible mistakes here, please correct me.
For instance, we include a Cython kD-tree that we use to provide a nearest-neighbor search when doing merger trees.
Thanks, I will take a look into that!
and you didn't want to go full-on "lagrangian coherent structures"
I took a look at it and it seems really interesting. It's really a pity I don't have enough time at the moment.
best wishes, Patrick
On 2012-10-05 18:07, Matthew Turk wrote:
Hi Patrick,
Thanks for writing, and welcome to yt-users. :)
On Thu, Oct 4, 2012 at 9:42 AM, Patrick Rieser <patrick.rieser@uibk.ac.at> wrote:
Heya everyone!
I am trying to track clumps across multiple snapshots (from flash). Now my questions is, has anybody done something like this and would be willing to share his/her code?
David Collins wrote some code that did this, a couple years ago. I don't know the current status.
If you wanted to write a new set of code to do this (and you didn't want to go full-on "lagrangian coherent structures") there are some things in yt that could help out. For instance, we include a Cython kD-tree that we use to provide a nearest-neighbor search when doing merger trees. This is used in a very simple way in the code in yt/analysis_modules/halo_merger_tree/enzofof_merger_tree.py , where halos are loaded into a variable called halo_kdtree. This then gets searched with a ball query. You could in principle load the clumps into the same kdtree structure, perform the search, and then apply selection criteria for clump tracking based on that. (Of course this is just the first step in identifying clump motion -- but it would be a way to reduce from N^2 searching.)
Let us know if you run into any tricks or have any successes -- this is a pretty cool idea, and I'd love to see where it leads you!
-Matt
Best Wishes, Patrick _______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
-- Sent from my computer.
participants (4)
-
David Collins
-
Eve Lee
-
Matthew Turk
-
Patrick Rieser