Hi Junhwan,

This is a fundamental failing with the way yt 3.0 and 3.1 deals with particles. I have reworked this in an open pull request, which is experimental (but nearing completion, prior to review) if you would like to try it out; it should make it both feasible and considerably faster. Let me know off list if you're willing to try it out.

-Matt

On Thu, Apr 9, 2015 at 2:54 PM, Junhwan Choi (최준환) choi.junhwan@gmail.com wrote:

Hi yt users and developer,

I tried to make a simple visualization for 244/h cMpc box with 512^3 DM and gas gadget simulation with following script ================================== import matplotlib as matplotlib matplotlib.use('Agg') import matplotlib.pylab as pylab import numpy as np import yt import yt.units as units

index = 0 text_dict = {'color':'white', 'size':20} fname = ('../snap_vbc_%03d.hdf5' % index) unit_base = {'UnitLength_in_cm' : 3.08568e+21, 'UnitMass_in_g' : 1.989e+43, 'UnitVelocity_in_cm_per_s' : 100000} bbox_lim = 244000 #kpc bbox = [[0,bbox_lim], [0,bbox_lim], [0,bbox_lim]]

ds = yt.load(fname,unit_base=unit_base,bounding_box=bbox) cen_center=np.array([122000,122000,122000])

pc = yt.ProjectionPlot(ds, "z", ('deposit', 'PartType0_density'), width = (244.0, 'Mpccm/h')) pc.annotate_text((0.8,0.9),'z=%4.2f' % ds.current_redshift, data_coords=False, text_args = text_dict) pc.save("all_view_%04d" % index)

pc = yt.ProjectionPlot(ds, "z", ('deposit', 'PartType1_density'), width = (244.0, 'Mpccm/h')) pc.annotate_text((0.8,0.9),'z=%4.2f' % ds.current_redshift, data_coords=False, text_args = text_dict) pc.save("all_view_%04d" % index)

pc = yt.ProjectionPlot(ds, "z", ('deposit', 'all_density'), width = (244.0, 'Mpccm/h')) pc.annotate_text((0.8,0.9),'z=%4.2f' % ds.current_redshift, data_coords=False, text_args = text_dict) pc.save("all_view_%04d" % index)

pc = yt.ProjectionPlot(ds, "z", ('gas', 'density'), width = (244.0, 'Mpccm/h')) pc.annotate_text((0.8,0.9),'z=%4.2f' % ds.current_redshift, data_coords=False, text_args = text_dict) pc.save("all_view_%04d" % index) ===================================

It initially works an make couple figures but it crash with following error message: ....... Traceback (most recent call last): File "viz.py", line 56, in <module> viz4(index) File "viz.py", line 50, in viz4 pc = yt.ProjectionPlot(ds, "z", ('gas', 'density'), width = (244.0, 'Mpccm/h')) File "/work/00916/tg459470/common/anaconda/lib/python2.7/site-packages/yt-3.2dev-py2.7-linux-x86_64.egg/yt/visualization/plot_window.py", line 1228, in __init__ field_parameters = field_parameters, method = method) File "/work/00916/tg459470/common/anaconda/lib/python2.7/site-packages/yt-3.2dev-py2.7-linux-x86_64.egg/yt/data_objects/construction_data_containers.py", line 250, in __init__ self.get_data(field) File "/work/00916/tg459470/common/anaconda/lib/python2.7/site-packages/yt-3.2dev-py2.7-linux-x86_64.egg/yt/data_objects/construction_data_containers.py", line 295, in get_data chunk_fields, "io", local_only = True)): File "/work/00916/tg459470/common/anaconda/lib/python2.7/site-packages/yt-3.2dev-py2.7-linux-x86_64.egg/yt/utilities/parallel_tools/parallel_analysis_interface.py", line 507, in para llel_objects for obj_id, obj in oiter: File "/work/00916/tg459470/common/anaconda/lib/python2.7/site-packages/yt-3.2dev-py2.7-linux-x86_64.egg/yt/data_objects/data_containers.py", line 592, in chunks self.get_data(fields) File "/work/00916/tg459470/common/anaconda/lib/python2.7/site-packages/yt-3.2dev-py2.7-linux-x86_64.egg/yt/data_objects/data_containers.py", line 693, in get_data self._generate_fields(fields_to_generate) File "/work/00916/tg459470/common/anaconda/lib/python2.7/site-packages/yt-3.2dev-py2.7-linux-x86_64.egg/yt/data_objects/data_containers.py", line 713, in _generate_fields fd = self._generate_field(field) File "/work/00916/tg459470/common/anaconda/lib/python2.7/site-packages/yt-3.2dev-py2.7-linux-x86_64.egg/yt/data_objects/data_containers.py", line 285, in _generate_field tr = self._generate_fluid_field(field) File "/work/00916/tg459470/common/anaconda/lib/python2.7/site-packages/yt-3.2dev-py2.7-linux-x86_64.egg/yt/data_objects/data_containers.py", line 303, in _generate_fluid_field rv = self._generate_spatial_fluid(field, ngt_exception.ghost_zones) File "/work/00916/tg459470/common/anaconda/lib/python2.7/site-packages/yt-3.2dev-py2.7-linux-x86_64.egg/yt/data_objects/data_containers.py", line 319, in _generate_spatial_fluid ind += o.select(self.selector, self[field], rv, ind) File "/work/00916/tg459470/common/anaconda/lib/python2.7/site-packages/yt-3.2dev-py2.7-linux-x86_64.egg/yt/data_objects/data_containers.py", line 248, in __getitem__ self.get_data(f) File "/work/00916/tg459470/common/anaconda/lib/python2.7/site-packages/yt-3.2dev-py2.7-linux-x86_64.egg/yt/data_objects/data_containers.py", line 693, in get_data self._generate_fields(fields_to_generate) File "/work/00916/tg459470/common/anaconda/lib/python2.7/site-packages/yt-3.2dev-py2.7-linux-x86_64.egg/yt/data_objects/data_containers.py", line 713, in _generate_fields fd = self._generate_field(field) File "/work/00916/tg459470/common/anaconda/lib/python2.7/site-packages/yt-3.2dev-py2.7-linux-x86_64.egg/yt/data_objects/data_containers.py", line 285, in _generate_field tr = self._generate_fluid_field(field) File "/work/00916/tg459470/common/anaconda/lib/python2.7/site-packages/yt-3.2dev-py2.7-linux-x86_64.egg/yt/data_objects/data_containers.py", line 305, in _generate_fluid_field rv = finfo(gen_obj) File "/work/00916/tg459470/common/anaconda/lib/python2.7/site-packages/yt-3.2dev-py2.7-linux-x86_64.egg/yt/fields/derived_field.py", line 184, in __call__ dd = self._function(self, data) File "/work/00916/tg459470/common/anaconda/lib/python2.7/site-packages/yt-3.2dev-py2.7-linux-x86_64.egg/yt/fields/particle_fields.py", line 958, in _vol_weight dens = data[ptype, density_name].in_cgs() File "/work/00916/tg459470/common/anaconda/lib/python2.7/site-packages/yt-3.2dev-py2.7-linux-x86_64.egg/yt/data_objects/octree_subset.py", line 77, in __getitem__ tr = super(OctreeSubset, self).__getitem__(key) File "/work/00916/tg459470/common/anaconda/lib/python2.7/site-packages/yt-3.2dev-py2.7-linux-x86_64.egg/yt/data_objects/data_containers.py", line 248, in __getitem__ self.get_data(f) File "/work/00916/tg459470/common/anaconda/lib/python2.7/site-packages/yt-3.2dev-py2.7-linux-x86_64.egg/yt/data_objects/data_containers.py", line 687, in get_data particles, self, self._current_chunk) File "/work/00916/tg459470/common/anaconda/lib/python2.7/site-packages/yt-3.2dev-py2.7-linux-x86_64.egg/yt/geometry/geometry_handler.py", line 234, in _read_particle_fields fields_to_read) File "/work/00916/tg459470/common/anaconda/lib/python2.7/site-packages/yt-3.2dev-py2.7-linux-x86_64.egg/yt/utilities/io_handler.py", line 177, in _read_particle_selection for field_r, vals in self._read_particle_fields(chunks, ptf, selector): File "/work/00916/tg459470/common/anaconda/lib/python2.7/site-packages/yt-3.2dev-py2.7-linux-x86_64.egg/yt/frontends/owls/io.py", line 97, in _read_particle_fields coords = g["Coordinates"][:].astype("float64") MemoryError

It looks there is a memory shortage. I use the stampede normal node which has ~32GB and I do not think 512^3 gas and DM particles require this amount memory. [But, I successfully made a visualization with 256^3 in 100/h cMpc run.] Is there any my possible mistake that leaks the memory? If so, could you inform me to overcome this memory issue.

Thank you in advance, Junhwan Choi _______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org

_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org