How to efficiently control the memory in yt with large simulation?
Hi all,
I try to make some visualizations (density/temperature project) for large simulation with yt. The simulation is 4096^3 unigrid Ramses simulation.
I try to implement the basics density and temperature projection plot with following script and I got memory problem and yt run is crashed. ===== from yt.mods import *
ds = load("../output_00048/info_00048.txt", fields = ["Density","x-velocity", "y-velocity","z-velocity","Pressure","Metallicity","Rad"]) center = [0., 0., 0.]
pw = ProjectionPlot(ds, "x", ("gas", "Density"), weight_field="Density", center=center) pw.zoom(1.01) pw.save("allviewGas") pw = ProjectionPlot(ds, "y", ("gas", "Density"), weight_field="Density", center=center) pw.zoom(1.01) pw.save("allviewGas") pw = ProjectionPlot(ds, "z", ("gas", "Density") , weight_field="Density", center=center) pw.zoom(1.01) pw.save("allviewGas")
pw = ProjectionPlot(ds, "x", "Temperature", weight_field="Density", center=center) pw.zoom(1.01) pw.save("allviewGas") pw = ProjectionPlot(ds, "y", "Temperature", weight_field="Density", center=center) pw.zoom(1.01) pw.save("allviewGas") pw = ProjectionPlot(ds, "z", "Temperature" , weight_field="Density", center=center) pw.zoom(1.01) pw.save("allviewGas") ====
Is there any way to reduce the memory usage when I make visualization? If I use slice instead of projection, can I save the memory? I saw there is parallelization for yt. Does parallelization distribute the data at the beginning of the read? (But, I prefer to do so w/o parallelization at this moment.)
Thank you in advance, Junhwan
Hi Junhwan,
There's some more background on this issue here:
http://lists.spacepope.org/pipermail/yt-users-spacepope.org/2013-November/00...
Basically, what it amounts to is:
* Right now RAMSES uses too much memory and duplicates the full mesh on every processor * This is not how it will always be * Unfortunately changing this can't be prioritized in the next couple weeks
Your mesh is *particularly* large for what we've dealt with before. As it stands, I would be surprised if it will work. The fix for this would be to change the Octrees to only parse on demand rather than at instantiation of the RAMSESHierarchy. Sam Geen and I have talked a bit about this, and he may be interested in working on it. It's a change that also will happen for N-body datasets (differently) and it's definitely planned to come, but it isn't being prioritized at this very moment because of other pressing concerns.
On Thu, Dec 5, 2013 at 10:47 PM, Junhwan Choi (최준환) choi.junhwan@gmail.com wrote:
Hi all,
I try to make some visualizations (density/temperature project) for large simulation with yt. The simulation is 4096^3 unigrid Ramses simulation.
I try to implement the basics density and temperature projection plot with following script and I got memory problem and yt run is crashed. ===== from yt.mods import *
ds = load("../output_00048/info_00048.txt", fields = ["Density","x-velocity", "y-velocity","z-velocity","Pressure","Metallicity","Rad"]) center = [0., 0., 0.]
pw = ProjectionPlot(ds, "x", ("gas", "Density"), weight_field="Density", center=center) pw.zoom(1.01) pw.save("allviewGas") pw = ProjectionPlot(ds, "y", ("gas", "Density"), weight_field="Density", center=center) pw.zoom(1.01) pw.save("allviewGas") pw = ProjectionPlot(ds, "z", ("gas", "Density") , weight_field="Density", center=center) pw.zoom(1.01) pw.save("allviewGas")
pw = ProjectionPlot(ds, "x", "Temperature", weight_field="Density", center=center) pw.zoom(1.01) pw.save("allviewGas") pw = ProjectionPlot(ds, "y", "Temperature", weight_field="Density", center=center) pw.zoom(1.01) pw.save("allviewGas") pw = ProjectionPlot(ds, "z", "Temperature" , weight_field="Density", center=center) pw.zoom(1.01) pw.save("allviewGas") ====
Is there any way to reduce the memory usage when I make visualization? If I use slice instead of projection, can I save the memory? I saw there is parallelization for yt. Does parallelization distribute the data at the beginning of the read? (But, I prefer to do so w/o parallelization at this moment.)
In principle, yes, slices will considerably reduce the memory, modulo the overhead from having all your octrees in memory at once -- which will be considerable.
I will spend some time thinking if there's a hotfix we can apply to make this work for you right now, as is, but I suspect it may be a little time until it can be properly implemented.
-Matt
Thank you in advance, Junhwan _______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
One possibility would be to define a bounding box that yt uses to upload ramses data only when contained in this bounding box. This can be done efficiently using the Hilbert key.
Romain
On 06 Dec 2013, at 15:49, Matthew Turk matthewturk@gmail.com wrote:
Hi Junhwan,
There's some more background on this issue here:
http://lists.spacepope.org/pipermail/yt-users-spacepope.org/2013-November/00...
Basically, what it amounts to is:
- Right now RAMSES uses too much memory and duplicates the full mesh
on every processor
- This is not how it will always be
- Unfortunately changing this can't be prioritized in the next couple weeks
Your mesh is *particularly* large for what we've dealt with before. As it stands, I would be surprised if it will work. The fix for this would be to change the Octrees to only parse on demand rather than at instantiation of the RAMSESHierarchy. Sam Geen and I have talked a bit about this, and he may be interested in working on it. It's a change that also will happen for N-body datasets (differently) and it's definitely planned to come, but it isn't being prioritized at this very moment because of other pressing concerns.
On Thu, Dec 5, 2013 at 10:47 PM, Junhwan Choi (최준환) choi.junhwan@gmail.com wrote:
Hi all,
I try to make some visualizations (density/temperature project) for large simulation with yt. The simulation is 4096^3 unigrid Ramses simulation.
I try to implement the basics density and temperature projection plot with following script and I got memory problem and yt run is crashed. ===== from yt.mods import *
ds = load("../output_00048/info_00048.txt", fields = ["Density","x-velocity", "y-velocity","z-velocity","Pressure","Metallicity","Rad"]) center = [0., 0., 0.]
pw = ProjectionPlot(ds, "x", ("gas", "Density"), weight_field="Density", center=center) pw.zoom(1.01) pw.save("allviewGas") pw = ProjectionPlot(ds, "y", ("gas", "Density"), weight_field="Density", center=center) pw.zoom(1.01) pw.save("allviewGas") pw = ProjectionPlot(ds, "z", ("gas", "Density") , weight_field="Density", center=center) pw.zoom(1.01) pw.save("allviewGas")
pw = ProjectionPlot(ds, "x", "Temperature", weight_field="Density", center=center) pw.zoom(1.01) pw.save("allviewGas") pw = ProjectionPlot(ds, "y", "Temperature", weight_field="Density", center=center) pw.zoom(1.01) pw.save("allviewGas") pw = ProjectionPlot(ds, "z", "Temperature" , weight_field="Density", center=center) pw.zoom(1.01) pw.save("allviewGas") ====
Is there any way to reduce the memory usage when I make visualization? If I use slice instead of projection, can I save the memory? I saw there is parallelization for yt. Does parallelization distribute the data at the beginning of the read? (But, I prefer to do so w/o parallelization at this moment.)
In principle, yes, slices will considerably reduce the memory, modulo the overhead from having all your octrees in memory at once -- which will be considerable.
I will spend some time thinking if there's a hotfix we can apply to make this work for you right now, as is, but I suspect it may be a little time until it can be properly implemented.
-Matt
Thank you in advance, Junhwan _______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
On Fri, Dec 6, 2013 at 9:54 AM, Romain Teyssier romain.teyssier@gmail.com wrote:
One possibility would be to define a bounding box that yt uses to upload ramses data only when contained in this bounding box. This can be done efficiently using the Hilbert key.
Romain
That would be good, and we could probably port the code Doug Rudd wrote to use SFC logic for detecting which zones to parse over to RAMSES, as well.
Today is out for me on this, because of other things, but with this I think we might be able to make it work in the near future, like mid-next week, assuming all goes well otherwise.
-Matt
On 06 Dec 2013, at 15:49, Matthew Turk matthewturk@gmail.com wrote:
Hi Junhwan,
There's some more background on this issue here:
http://lists.spacepope.org/pipermail/yt-users-spacepope.org/2013-November/00...
Basically, what it amounts to is:
- Right now RAMSES uses too much memory and duplicates the full mesh
on every processor
- This is not how it will always be
- Unfortunately changing this can't be prioritized in the next couple weeks
Your mesh is *particularly* large for what we've dealt with before. As it stands, I would be surprised if it will work. The fix for this would be to change the Octrees to only parse on demand rather than at instantiation of the RAMSESHierarchy. Sam Geen and I have talked a bit about this, and he may be interested in working on it. It's a change that also will happen for N-body datasets (differently) and it's definitely planned to come, but it isn't being prioritized at this very moment because of other pressing concerns.
On Thu, Dec 5, 2013 at 10:47 PM, Junhwan Choi (최준환) choi.junhwan@gmail.com wrote:
Hi all,
I try to make some visualizations (density/temperature project) for large simulation with yt. The simulation is 4096^3 unigrid Ramses simulation.
I try to implement the basics density and temperature projection plot with following script and I got memory problem and yt run is crashed. ===== from yt.mods import *
ds = load("../output_00048/info_00048.txt", fields = ["Density","x-velocity", "y-velocity","z-velocity","Pressure","Metallicity","Rad"]) center = [0., 0., 0.]
pw = ProjectionPlot(ds, "x", ("gas", "Density"), weight_field="Density", center=center) pw.zoom(1.01) pw.save("allviewGas") pw = ProjectionPlot(ds, "y", ("gas", "Density"), weight_field="Density", center=center) pw.zoom(1.01) pw.save("allviewGas") pw = ProjectionPlot(ds, "z", ("gas", "Density") , weight_field="Density", center=center) pw.zoom(1.01) pw.save("allviewGas")
pw = ProjectionPlot(ds, "x", "Temperature", weight_field="Density", center=center) pw.zoom(1.01) pw.save("allviewGas") pw = ProjectionPlot(ds, "y", "Temperature", weight_field="Density", center=center) pw.zoom(1.01) pw.save("allviewGas") pw = ProjectionPlot(ds, "z", "Temperature" , weight_field="Density", center=center) pw.zoom(1.01) pw.save("allviewGas") ====
Is there any way to reduce the memory usage when I make visualization? If I use slice instead of projection, can I save the memory? I saw there is parallelization for yt. Does parallelization distribute the data at the beginning of the read? (But, I prefer to do so w/o parallelization at this moment.)
In principle, yes, slices will considerably reduce the memory, modulo the overhead from having all your octrees in memory at once -- which will be considerable.
I will spend some time thinking if there's a hotfix we can apply to make this work for you right now, as is, but I suspect it may be a little time until it can be properly implemented.
-Matt
Thank you in advance, Junhwan _______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
Thank you Matt and Romain,
Now, I understand it is a more fundamental issue in the yt with Ramses, and I need to wait yt to be updated. Meanwhile, is there any way I can access small fraction of the simulation data (such as thin slice of the data) and make some visualization plots such as thin projection and slice? If so, I can reduce the required memory. Is it possible?
Thank you, Junhwan
On Fri, Dec 6, 2013 at 8:56 AM, Matthew Turk matthewturk@gmail.com wrote:
On Fri, Dec 6, 2013 at 9:54 AM, Romain Teyssier romain.teyssier@gmail.com wrote:
One possibility would be to define a bounding box that yt uses to upload ramses data only when contained in this bounding box. This can be done efficiently using the Hilbert key.
Romain
That would be good, and we could probably port the code Doug Rudd wrote to use SFC logic for detecting which zones to parse over to RAMSES, as well.
Today is out for me on this, because of other things, but with this I think we might be able to make it work in the near future, like mid-next week, assuming all goes well otherwise.
-Matt
On 06 Dec 2013, at 15:49, Matthew Turk matthewturk@gmail.com wrote:
Hi Junhwan,
There's some more background on this issue here:
http://lists.spacepope.org/pipermail/yt-users-spacepope.org/2013-November/00...
Basically, what it amounts to is:
- Right now RAMSES uses too much memory and duplicates the full mesh
on every processor
- This is not how it will always be
- Unfortunately changing this can't be prioritized in the next couple weeks
Your mesh is *particularly* large for what we've dealt with before. As it stands, I would be surprised if it will work. The fix for this would be to change the Octrees to only parse on demand rather than at instantiation of the RAMSESHierarchy. Sam Geen and I have talked a bit about this, and he may be interested in working on it. It's a change that also will happen for N-body datasets (differently) and it's definitely planned to come, but it isn't being prioritized at this very moment because of other pressing concerns.
On Thu, Dec 5, 2013 at 10:47 PM, Junhwan Choi (최준환) choi.junhwan@gmail.com wrote:
Hi all,
I try to make some visualizations (density/temperature project) for large simulation with yt. The simulation is 4096^3 unigrid Ramses simulation.
I try to implement the basics density and temperature projection plot with following script and I got memory problem and yt run is crashed. ===== from yt.mods import *
ds = load("../output_00048/info_00048.txt", fields = ["Density","x-velocity", "y-velocity","z-velocity","Pressure","Metallicity","Rad"]) center = [0., 0., 0.]
pw = ProjectionPlot(ds, "x", ("gas", "Density"), weight_field="Density", center=center) pw.zoom(1.01) pw.save("allviewGas") pw = ProjectionPlot(ds, "y", ("gas", "Density"), weight_field="Density", center=center) pw.zoom(1.01) pw.save("allviewGas") pw = ProjectionPlot(ds, "z", ("gas", "Density") , weight_field="Density", center=center) pw.zoom(1.01) pw.save("allviewGas")
pw = ProjectionPlot(ds, "x", "Temperature", weight_field="Density", center=center) pw.zoom(1.01) pw.save("allviewGas") pw = ProjectionPlot(ds, "y", "Temperature", weight_field="Density", center=center) pw.zoom(1.01) pw.save("allviewGas") pw = ProjectionPlot(ds, "z", "Temperature" , weight_field="Density", center=center) pw.zoom(1.01) pw.save("allviewGas") ====
Is there any way to reduce the memory usage when I make visualization? If I use slice instead of projection, can I save the memory? I saw there is parallelization for yt. Does parallelization distribute the data at the beginning of the read? (But, I prefer to do so w/o parallelization at this moment.)
In principle, yes, slices will considerably reduce the memory, modulo the overhead from having all your octrees in memory at once -- which will be considerable.
I will spend some time thinking if there's a hotfix we can apply to make this work for you right now, as is, but I suspect it may be a little time until it can be properly implemented.
-Matt
Thank you in advance, Junhwan _______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
I believe you can access rectangular sub volume of the data with something like
sv = pf.h.region([0.5]*3, [0.21, .21, .72], [.28, .28, .79])
Shown here when running halo finder on a sub volume http://yt-project.org/docs/2.6/analyzing/analysis_modules/running_halofinder...
And you can use other 3D objects like spheres, ellipsoids etc, for a thin slice of the data you can make your rectangular region small in one of the dimension.
When making plots just pass in the sv 3D sub volume object instead of the usual pf for plotting the entire simulation.
Hope that helps.
From
G.S.
On Fri, Dec 6, 2013 at 8:41 AM, Junhwan Choi (최준환) choi.junhwan@gmail.comwrote:
Thank you Matt and Romain,
Now, I understand it is a more fundamental issue in the yt with Ramses, and I need to wait yt to be updated. Meanwhile, is there any way I can access small fraction of the simulation data (such as thin slice of the data) and make some visualization plots such as thin projection and slice? If so, I can reduce the required memory. Is it possible?
Thank you, Junhwan
On Fri, Dec 6, 2013 at 8:56 AM, Matthew Turk matthewturk@gmail.com wrote:
On Fri, Dec 6, 2013 at 9:54 AM, Romain Teyssier romain.teyssier@gmail.com wrote:
One possibility would be to define a bounding box that yt uses to upload ramses data only when contained in this bounding box. This can be done efficiently using the Hilbert key.
Romain
That would be good, and we could probably port the code Doug Rudd wrote to use SFC logic for detecting which zones to parse over to RAMSES, as well.
Today is out for me on this, because of other things, but with this I think we might be able to make it work in the near future, like mid-next week, assuming all goes well otherwise.
-Matt
On 06 Dec 2013, at 15:49, Matthew Turk matthewturk@gmail.com wrote:
Hi Junhwan,
There's some more background on this issue here:
http://lists.spacepope.org/pipermail/yt-users-spacepope.org/2013-November/00...
Basically, what it amounts to is:
- Right now RAMSES uses too much memory and duplicates the full mesh
on every processor
- This is not how it will always be
- Unfortunately changing this can't be prioritized in the next couple
weeks
Your mesh is *particularly* large for what we've dealt with before. As it stands, I would be surprised if it will work. The fix for this would be to change the Octrees to only parse on demand rather than at instantiation of the RAMSESHierarchy. Sam Geen and I have talked a bit about this, and he may be interested in working on it. It's a change that also will happen for N-body datasets (differently) and it's definitely planned to come, but it isn't being prioritized at this very moment because of other pressing concerns.
On Thu, Dec 5, 2013 at 10:47 PM, Junhwan Choi (최준환) choi.junhwan@gmail.com wrote:
Hi all,
I try to make some visualizations (density/temperature project) for large simulation with yt. The simulation is 4096^3 unigrid Ramses simulation.
I try to implement the basics density and temperature projection plot with following script and I got memory problem and yt run is crashed. ===== from yt.mods import *
ds = load("../output_00048/info_00048.txt", fields = ["Density","x-velocity", "y-velocity","z-velocity","Pressure","Metallicity","Rad"]) center = [0., 0., 0.]
pw = ProjectionPlot(ds, "x", ("gas", "Density"), weight_field="Density", center=center) pw.zoom(1.01) pw.save("allviewGas") pw = ProjectionPlot(ds, "y", ("gas", "Density"), weight_field="Density", center=center) pw.zoom(1.01) pw.save("allviewGas") pw = ProjectionPlot(ds, "z", ("gas", "Density") , weight_field="Density", center=center) pw.zoom(1.01) pw.save("allviewGas")
pw = ProjectionPlot(ds, "x", "Temperature", weight_field="Density", center=center) pw.zoom(1.01) pw.save("allviewGas") pw = ProjectionPlot(ds, "y", "Temperature", weight_field="Density", center=center) pw.zoom(1.01) pw.save("allviewGas") pw = ProjectionPlot(ds, "z", "Temperature" , weight_field="Density", center=center) pw.zoom(1.01) pw.save("allviewGas") ====
Is there any way to reduce the memory usage when I make visualization? If I use slice instead of projection, can I save the memory? I saw there is parallelization for yt. Does parallelization distribute the data at the beginning of the read? (But, I prefer to do so w/o parallelization at this moment.)
In principle, yes, slices will considerably reduce the memory, modulo the overhead from having all your octrees in memory at once -- which will be considerable.
I will spend some time thinking if there's a hotfix we can apply to make this work for you right now, as is, but I suspect it may be a little time until it can be properly implemented.
-Matt
Thank you in advance, Junhwan _______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
On Fri, Dec 6, 2013 at 2:52 PM, Geoffrey So gsiisg@gmail.com wrote:
I believe you can access rectangular sub volume of the data with something like
sv = pf.h.region([0.5]*3, [0.21, .21, .72], [.28, .28, .79])
Shown here when running halo finder on a sub volume http://yt-project.org/docs/2.6/analyzing/analysis_modules/running_halofinder...
And you can use other 3D objects like spheres, ellipsoids etc, for a thin slice of the data you can make your rectangular region small in one of the dimension.
When making plots just pass in the sv 3D sub volume object instead of the usual pf for plotting the entire simulation.
This will work if the Octree is parsed on demand; what I think Junhwan is running into right now is that the entire octree is loaded at startup. So you end up with a memory constant that gets added on for each process; this will reduce the *additional* overhead of loading data, but he still has memory for each leaf node of the oct.
Hope that helps.
From
G.S.
On Fri, Dec 6, 2013 at 8:41 AM, Junhwan Choi (최준환) choi.junhwan@gmail.com wrote:
Thank you Matt and Romain,
Now, I understand it is a more fundamental issue in the yt with Ramses, and I need to wait yt to be updated. Meanwhile, is there any way I can access small fraction of the simulation data (such as thin slice of the data) and make some visualization plots such as thin projection and slice? If so, I can reduce the required memory. Is it possible?
Thank you, Junhwan
On Fri, Dec 6, 2013 at 8:56 AM, Matthew Turk matthewturk@gmail.com wrote:
On Fri, Dec 6, 2013 at 9:54 AM, Romain Teyssier romain.teyssier@gmail.com wrote:
One possibility would be to define a bounding box that yt uses to upload ramses data only when contained in this bounding box. This can be done efficiently using the Hilbert key.
Romain
That would be good, and we could probably port the code Doug Rudd wrote to use SFC logic for detecting which zones to parse over to RAMSES, as well.
Today is out for me on this, because of other things, but with this I think we might be able to make it work in the near future, like mid-next week, assuming all goes well otherwise.
-Matt
On 06 Dec 2013, at 15:49, Matthew Turk matthewturk@gmail.com wrote:
Hi Junhwan,
There's some more background on this issue here:
http://lists.spacepope.org/pipermail/yt-users-spacepope.org/2013-November/00...
Basically, what it amounts to is:
- Right now RAMSES uses too much memory and duplicates the full mesh
on every processor
- This is not how it will always be
- Unfortunately changing this can't be prioritized in the next couple
weeks
Your mesh is *particularly* large for what we've dealt with before. As it stands, I would be surprised if it will work. The fix for this would be to change the Octrees to only parse on demand rather than at instantiation of the RAMSESHierarchy. Sam Geen and I have talked a bit about this, and he may be interested in working on it. It's a change that also will happen for N-body datasets (differently) and it's definitely planned to come, but it isn't being prioritized at this very moment because of other pressing concerns.
On Thu, Dec 5, 2013 at 10:47 PM, Junhwan Choi (최준환) choi.junhwan@gmail.com wrote:
Hi all,
I try to make some visualizations (density/temperature project) for large simulation with yt. The simulation is 4096^3 unigrid Ramses simulation.
I try to implement the basics density and temperature projection plot with following script and I got memory problem and yt run is crashed. ===== from yt.mods import *
ds = load("../output_00048/info_00048.txt", fields = ["Density","x-velocity", "y-velocity","z-velocity","Pressure","Metallicity","Rad"]) center = [0., 0., 0.]
pw = ProjectionPlot(ds, "x", ("gas", "Density"), weight_field="Density", center=center) pw.zoom(1.01) pw.save("allviewGas") pw = ProjectionPlot(ds, "y", ("gas", "Density"), weight_field="Density", center=center) pw.zoom(1.01) pw.save("allviewGas") pw = ProjectionPlot(ds, "z", ("gas", "Density") , weight_field="Density", center=center) pw.zoom(1.01) pw.save("allviewGas")
pw = ProjectionPlot(ds, "x", "Temperature", weight_field="Density", center=center) pw.zoom(1.01) pw.save("allviewGas") pw = ProjectionPlot(ds, "y", "Temperature", weight_field="Density", center=center) pw.zoom(1.01) pw.save("allviewGas") pw = ProjectionPlot(ds, "z", "Temperature" , weight_field="Density", center=center) pw.zoom(1.01) pw.save("allviewGas") ====
Is there any way to reduce the memory usage when I make visualization? If I use slice instead of projection, can I save the memory? I saw there is parallelization for yt. Does parallelization distribute the data at the beginning of the read? (But, I prefer to do so w/o parallelization at this moment.)
In principle, yes, slices will considerably reduce the memory, modulo the overhead from having all your octrees in memory at once -- which will be considerable.
I will spend some time thinking if there's a hotfix we can apply to make this work for you right now, as is, but I suspect it may be a little time until it can be properly implemented.
-Matt
Thank you in advance, Junhwan _______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
On Fri, Dec 6, 2013 at 2:52 PM, Geoffrey So gsiisg@gmail.com wrote:
I believe you can access rectangular sub volume of the data with something like
sv = pf.h.region([0.5]*3, [0.21, .21, .72], [.28, .28, .79])
Shown here when running halo finder on a sub volume http://yt-project.org/docs/2.6/analyzing/analysis_modules/running_halofinder...
And you can use other 3D objects like spheres, ellipsoids etc, for a thin slice of the data you can make your rectangular region small in one of the dimension.
When making plots just pass in the sv 3D sub volume object instead of the usual pf for plotting the entire simulation.
This will work if the Octree is parsed on demand; what I think Junhwan is running into right now is that the entire octree is loaded at startup. So you end up with a memory constant that gets added on for each process; this will reduce the *additional* overhead of loading data, but he still has memory for each leaf node of the oct.
Do you mean that there is no way to read 4098^3 Ramses data using yt? I tried to define small region of the simulation and make a density projection plot, but it looks fail owing to the same reason. Is there way to temporally get around this problem?
Thank you for continuous help, Junhwan
participants (4)
-
Geoffrey So -
Junhwan Choi (최준환) -
Matthew Turk -
Romain Teyssier