cloud in cell mesh construction for particle data
Hi, I was wondering if there were any Cython routines within yt which takes particle data and converts it into a cloud-in-cell based mesh which can be written to a file of my choosing. I heard a while ago there was some such functionality but it could be too far down the yt rabbit hole to be used as a standalone? Is this true? I have my own Python code for doing it but it just isn't fast enough and thought I'd ask the yt community if there were any wrapper tools available to boost the speed. Thanks. Brendan
Hi Brendan, Which version of yt are you using? If you're using 3.0, this is actually fairly easy. If you look in yt.fields.particle_fields.py, around line 85, you can see how this is done for the "particle_density" and "particle_mass" fields. Basically you can call a "deposit" method which takes the particle field quantity you want deposited and deposits it into cells. The underlying calculation is done using Cython, so it's fast. If you're using 2.x, then you can do the same thing, but it's not as straightforward. You can see how this works in yt.data_objects.universal_fields.py, around line 986, where the "particle_density" field is defined. Basically, it calls CICDeposit_3, which is in yt.utilities.lib.CICDeposit.pyx. Let me know if you need any more clarification. Best, John Z On Jun 5, 2014, at 6:07 PM, Brendan Griffen <brendan.f.griffen@gmail.com> wrote:
Hi,
I was wondering if there were any Cython routines within yt which takes particle data and converts it into a cloud-in-cell based mesh which can be written to a file of my choosing. I heard a while ago there was some such functionality but it could be too far down the yt rabbit hole to be used as a standalone? Is this true? I have my own Python code for doing it but it just isn't fast enough and thought I'd ask the yt community if there were any wrapper tools available to boost the speed.
Thanks. Brendan _______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
On Thu, Jun 5, 2014 at 3:36 PM, John ZuHone <jzuhone@gmail.com> wrote:
Hi Brendan,
Which version of yt are you using?
If you're using 3.0, this is actually fairly easy. If you look in yt.fields.particle_fields.py, around line 85, you can see how this is done for the "particle_density" and "particle_mass" fields. Basically you can call a "deposit" method which takes the particle field quantity you want deposited and deposits it into cells. The underlying calculation is done using Cython, so it's fast.
And you shouldn't ever actually need to call these "deposit" functions, since "deposit" is exposed as a field type for all datasets that contain particles. Here is a notebook that does this for Enzo AMR data: http://nbviewer.ipython.org/gist/ngoldbaum/5e19e4e6cc2bf330149c This dataset contains about a million particles and generates a CIC deposition for the whole domain in about 6 seconds from a cold start.
If you're using 2.x, then you can do the same thing, but it's not as straightforward. You can see how this works in yt.data_objects.universal_fields.py, around line 986, where the "particle_density" field is defined. Basically, it calls CICDeposit_3, which is in yt.utilities.lib.CICDeposit.pyx.
Let me know if you need any more clarification.
Best,
John Z
On Jun 5, 2014, at 6:07 PM, Brendan Griffen <brendan.f.griffen@gmail.com> wrote:
Hi,
I was wondering if there were any Cython routines within yt which takes particle data and converts it into a cloud-in-cell based mesh which can be written to a file of my choosing.
What sort of mesh were you looking for? yt will internally construct an octree if it is fed particle data. I'm not sure whether this octree can be saved to disk for later analysis. It's also possible to create a uniform resolution covering grid containing field data for a deposited quantity, which can be quite easily saved to disk in a number of ways.
I heard a while ago there was some such functionality but it could be too far down the yt rabbit hole to be used as a standalone? Is this true? I have my own Python code for doing it but it just isn't fast enough and thought I'd ask the yt community if there were any wrapper tools available to boost the speed.
Thanks. Brendan _______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
Thanks. I'll get the "bleeding edge" version first then try your suggestions. Though I want to return the NxNxN array and be able to write this mesh to a file. It is *only* using the cic part of yt and it should return the mesh to be written? Just wanted to clarify? Thanks. Brendan On Thu, Jun 5, 2014 at 6:49 PM, Nathan Goldbaum <nathan12343@gmail.com> wrote:
On Thu, Jun 5, 2014 at 3:36 PM, John ZuHone <jzuhone@gmail.com> wrote:
Hi Brendan,
Which version of yt are you using?
If you're using 3.0, this is actually fairly easy. If you look in yt.fields.particle_fields.py, around line 85, you can see how this is done for the "particle_density" and "particle_mass" fields. Basically you can call a "deposit" method which takes the particle field quantity you want deposited and deposits it into cells. The underlying calculation is done using Cython, so it's fast.
And you shouldn't ever actually need to call these "deposit" functions, since "deposit" is exposed as a field type for all datasets that contain particles.
Here is a notebook that does this for Enzo AMR data:
http://nbviewer.ipython.org/gist/ngoldbaum/5e19e4e6cc2bf330149c
This dataset contains about a million particles and generates a CIC deposition for the whole domain in about 6 seconds from a cold start.
If you're using 2.x, then you can do the same thing, but it's not as straightforward. You can see how this works in yt.data_objects.universal_fields.py, around line 986, where the "particle_density" field is defined. Basically, it calls CICDeposit_3, which is in yt.utilities.lib.CICDeposit.pyx.
Let me know if you need any more clarification.
Best,
John Z
On Jun 5, 2014, at 6:07 PM, Brendan Griffen <brendan.f.griffen@gmail.com> wrote:
Hi,
I was wondering if there were any Cython routines within yt which takes particle data and converts it into a cloud-in-cell based mesh which can be written to a file of my choosing.
What sort of mesh were you looking for? yt will internally construct an octree if it is fed particle data. I'm not sure whether this octree can be saved to disk for later analysis.
It's also possible to create a uniform resolution covering grid containing field data for a deposited quantity, which can be quite easily saved to disk in a number of ways.
I heard a while ago there was some such functionality but it could be too far down the yt rabbit hole to be used as a standalone? Is this true? I have my own Python code for doing it but it just isn't fast enough and thought I'd ask the yt community if there were any wrapper tools available to boost the speed.
Thanks. Brendan _______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
Hey Brendan, A couple of extra tools you might find helpful in conjunction with Nathan's example of depositing the particles onto an octree are at: http://paste.yt-project.org/show/4737/ Where I load a gadget snapshot, and then recover the coordinates and width of each cell. In response to your last question - the particles are deposited into an octree grid (so, you'll see that the cell sizes aren't all the same size). I don't know if depositing onto a regular NxNxN mesh is possible, though would be interested to hear if so. -d On Thu, Jun 5, 2014 at 7:58 PM, Brendan Griffen <brendan.f.griffen@gmail.com
wrote:
Thanks. I'll get the "bleeding edge" version first then try your suggestions. Though I want to return the NxNxN array and be able to write this mesh to a file. It is *only* using the cic part of yt and it should return the mesh to be written? Just wanted to clarify?
Thanks. Brendan
On Thu, Jun 5, 2014 at 6:49 PM, Nathan Goldbaum <nathan12343@gmail.com> wrote:
On Thu, Jun 5, 2014 at 3:36 PM, John ZuHone <jzuhone@gmail.com> wrote:
Hi Brendan,
Which version of yt are you using?
If you're using 3.0, this is actually fairly easy. If you look in yt.fields.particle_fields.py, around line 85, you can see how this is done for the "particle_density" and "particle_mass" fields. Basically you can call a "deposit" method which takes the particle field quantity you want deposited and deposits it into cells. The underlying calculation is done using Cython, so it's fast.
And you shouldn't ever actually need to call these "deposit" functions, since "deposit" is exposed as a field type for all datasets that contain particles.
Here is a notebook that does this for Enzo AMR data:
http://nbviewer.ipython.org/gist/ngoldbaum/5e19e4e6cc2bf330149c
This dataset contains about a million particles and generates a CIC deposition for the whole domain in about 6 seconds from a cold start.
If you're using 2.x, then you can do the same thing, but it's not as straightforward. You can see how this works in yt.data_objects.universal_fields.py, around line 986, where the "particle_density" field is defined. Basically, it calls CICDeposit_3, which is in yt.utilities.lib.CICDeposit.pyx.
Let me know if you need any more clarification.
Best,
John Z
On Jun 5, 2014, at 6:07 PM, Brendan Griffen <brendan.f.griffen@gmail.com> wrote:
Hi,
I was wondering if there were any Cython routines within yt which takes particle data and converts it into a cloud-in-cell based mesh which can be written to a file of my choosing.
What sort of mesh were you looking for? yt will internally construct an octree if it is fed particle data. I'm not sure whether this octree can be saved to disk for later analysis.
It's also possible to create a uniform resolution covering grid containing field data for a deposited quantity, which can be quite easily saved to disk in a number of ways.
I heard a while ago there was some such functionality but it could be too far down the yt rabbit hole to be used as a standalone? Is this true? I have my own Python code for doing it but it just isn't fast enough and thought I'd ask the yt community if there were any wrapper tools available to boost the speed.
Thanks. Brendan _______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
Here's a worked out example that does what you're looking for using a fake 1 million particle dataset: http://nbviewer.ipython.org/gist/ngoldbaum/546d37869aafe71cfe38 In this notebook I make use of two key yt features: `load_particles`, and `covering_grid`. load_particles creates a "stream" dataset based on in-memory data fed in as a numpy array. This dataset acts just like an on-disk simulation dataset, but doesn't come with the baggage of needing to write a custom frontend to read a specific data format off disk. covering_grid is a way to generate uniform resolution data from an AMR dataset. It acts like a python dictionary where the keys are field names and returns 3D numpy arrays of whatever uniform resolution you specify when you create the covering_grid. Note that if you're using load_particles all of your data needs to live in memory. If your data is too big for that you'll need to write a frontend for your data format or use a memmap to an on-disk file somehow. I'm not an expert on that but others on the list should be able to help out. Hope that gets you well on your way :) -Nathan On Thu, Jun 5, 2014 at 5:04 PM, Desika Narayanan <dnarayan@haverford.edu> wrote:
Hey Brendan,
A couple of extra tools you might find helpful in conjunction with Nathan's example of depositing the particles onto an octree are at:
http://paste.yt-project.org/show/4737/
Where I load a gadget snapshot, and then recover the coordinates and width of each cell.
In response to your last question - the particles are deposited into an octree grid (so, you'll see that the cell sizes aren't all the same size). I don't know if depositing onto a regular NxNxN mesh is possible, though would be interested to hear if so.
-d
On Thu, Jun 5, 2014 at 7:58 PM, Brendan Griffen < brendan.f.griffen@gmail.com> wrote:
Thanks. I'll get the "bleeding edge" version first then try your suggestions. Though I want to return the NxNxN array and be able to write this mesh to a file. It is *only* using the cic part of yt and it should return the mesh to be written? Just wanted to clarify?
Thanks. Brendan
On Thu, Jun 5, 2014 at 6:49 PM, Nathan Goldbaum <nathan12343@gmail.com> wrote:
On Thu, Jun 5, 2014 at 3:36 PM, John ZuHone <jzuhone@gmail.com> wrote:
Hi Brendan,
Which version of yt are you using?
If you're using 3.0, this is actually fairly easy. If you look in yt.fields.particle_fields.py, around line 85, you can see how this is done for the "particle_density" and "particle_mass" fields. Basically you can call a "deposit" method which takes the particle field quantity you want deposited and deposits it into cells. The underlying calculation is done using Cython, so it's fast.
And you shouldn't ever actually need to call these "deposit" functions, since "deposit" is exposed as a field type for all datasets that contain particles.
Here is a notebook that does this for Enzo AMR data:
http://nbviewer.ipython.org/gist/ngoldbaum/5e19e4e6cc2bf330149c
This dataset contains about a million particles and generates a CIC deposition for the whole domain in about 6 seconds from a cold start.
If you're using 2.x, then you can do the same thing, but it's not as straightforward. You can see how this works in yt.data_objects.universal_fields.py, around line 986, where the "particle_density" field is defined. Basically, it calls CICDeposit_3, which is in yt.utilities.lib.CICDeposit.pyx.
Let me know if you need any more clarification.
Best,
John Z
On Jun 5, 2014, at 6:07 PM, Brendan Griffen < brendan.f.griffen@gmail.com> wrote:
Hi,
I was wondering if there were any Cython routines within yt which takes particle data and converts it into a cloud-in-cell based mesh which can be written to a file of my choosing.
What sort of mesh were you looking for? yt will internally construct an octree if it is fed particle data. I'm not sure whether this octree can be saved to disk for later analysis.
It's also possible to create a uniform resolution covering grid containing field data for a deposited quantity, which can be quite easily saved to disk in a number of ways.
I heard a while ago there was some such functionality but it could be too far down the yt rabbit hole to be used as a standalone? Is this true? I have my own Python code for doing it but it just isn't fast enough and thought I'd ask the yt community if there were any wrapper tools available to boost the speed.
Thanks. Brendan _______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
Thanks very much Nathan. I tried to load in my own data but I think there are too many particles or I have to specifically set the domain size. In this area: data = {'particle_position_x': pos[:,0], 'particle_position_y': pos[:,1], 'particle_position_z': pos[:,2], 'particle_mass': np.array([mpart]*npart)} ds = yt.load_particles(data, length_unit=3.08e24, mass_unit=1.9891e36) ad = ds.all_data() print ad['deposit', 'all_cic'] In [3]: run ytcic.py yt : [INFO ] 2014-06-05 21:29:06,183 Parameters: current_time = 0.0 yt : [INFO ] 2014-06-05 21:29:06,183 Parameters: domain_dimensions = [2 2 2] yt : [INFO ] 2014-06-05 21:29:06,184 Parameters: domain_left_edge = [ 0. 0. 0.] yt : [INFO ] 2014-06-05 21:29:06,185 Parameters: domain_right_edge = [ 1. 1. 1.] yt : [INFO ] 2014-06-05 21:29:06,185 Parameters: cosmological_simulation = 0.0 yt : [INFO ] 2014-06-05 21:29:06,188 Allocating for 1.342e+08 particles --------------------------------------------------------------------------- YTDomainOverflow Traceback (most recent call last) /nfs/blank/h4231/bgriffen/data/lib/yt-x86_64/lib/python2.7/site-packages/IPython/utils/py3compat.pyc in execfile(fname, *where) 202 else: 203 filename = fname --> 204 __builtin__.execfile(filename, *where) /nfs/blank/h4231/bgriffen/work/projects/caterpillar/c2ray/cic/ytcic.py in <module>() 52 53 ad = ds.all_data() ---> 54 print ad['deposit', 'all_cic'] 55 slc = yt.SlicePlot(ds, 2, ('deposit', 'all_cic')) 56 slc.set_figure_size(4) /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc in __getitem__(self, key) 205 Returns a single field. Will add if necessary. 206 """ --> 207 f = self._determine_fields([key])[0] 208 if f not in self.field_data and key not in self.field_data: 209 if f in self._container_fields: /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc in _determine_fields(self, fields) 453 raise YTFieldNotParseable(field) 454 ftype, fname = field --> 455 finfo = self.pf._get_field_info(ftype, fname) 456 else: 457 fname = field /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/static_output.pyc in _get_field_info(self, ftype, fname) 445 _last_finfo = None 446 def _get_field_info(self, ftype, fname = None): --> 447 self.index 448 if fname is None: 449 ftype, fname = "unknown", ftype /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/static_output.pyc in index(self) 277 raise RuntimeError("You should not instantiate Dataset.") 278 self._instantiated_index = self._index_class( --> 279 self, dataset_type=self.dataset_type) 280 # Now we do things that we need an instantiated index for 281 # ...first off, we create our field_info now. /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/frontends/stream/data_structures.pyc in __init__(self, pf, dataset_type) 942 def __init__(self, pf, dataset_type = None): 943 self.stream_handler = pf.stream_handler --> 944 super(StreamParticleIndex, self).__init__(pf, dataset_type) 945 946 def _setup_data_io(self): /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/geometry/particle_geometry_handler.pyc in __init__(self, pf, dataset_type) 48 self.directory = os.path.dirname(self.index_filename) 49 self.float_type = np.float64 ---> 50 super(ParticleIndex, self).__init__(pf, dataset_type) 51 52 def _setup_geometry(self): /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/geometry/geometry_handler.pyc in __init__(self, pf, dataset_type) 54 55 mylog.debug("Setting up domain geometry.") ---> 56 self._setup_geometry() 57 58 mylog.debug("Initializing data grid data IO") /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/geometry/particle_geometry_handler.pyc in _setup_geometry(self) 52 def _setup_geometry(self): 53 mylog.debug("Initializing Particle Geometry Handler.") ---> 54 self._initialize_particle_handler() 55 56 /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/geometry/particle_geometry_handler.pyc in _initialize_particle_handler(self) 87 pf.domain_left_edge, pf.domain_right_edge, 88 [N, N, N], len(self.data_files)) ---> 89 self._initialize_indices() 90 self.oct_handler.finalize() 91 self.max_level = self.oct_handler.max_level /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/geometry/particle_geometry_handler.pyc in _initialize_indices(self) 109 npart = sum(data_file.total_particles.values()) 110 morton[ind:ind + npart] = \ --> 111 self.io._initialize_index(data_file, self.regions) 112 ind += npart 113 morton.sort() /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/frontends/stream/io.pyc in _initialize_index(self, data_file, regions) 144 raise YTDomainOverflow(pos.min(axis=0), pos.max(axis=0), 145 data_file.pf.domain_left_edge, --> 146 data_file.pf.domain_right_edge) 147 regions.add_data_file(pos, data_file.file_id) 148 morton.append(compute_morton( YTDomainOverflow: Particle bounds [ 0. 0. 0.] and [ 99.99999237 99.99999237 99.99999237] exceed domain bounds [ 0. 0. 0.] code_length and [ 1. 1. 1.] code_length On Thu, Jun 5, 2014 at 8:22 PM, Nathan Goldbaum <nathan12343@gmail.com> wrote:
Here's a worked out example that does what you're looking for using a fake 1 million particle dataset:
http://nbviewer.ipython.org/gist/ngoldbaum/546d37869aafe71cfe38
In this notebook I make use of two key yt features: `load_particles`, and `covering_grid`.
load_particles creates a "stream" dataset based on in-memory data fed in as a numpy array. This dataset acts just like an on-disk simulation dataset, but doesn't come with the baggage of needing to write a custom frontend to read a specific data format off disk.
covering_grid is a way to generate uniform resolution data from an AMR dataset. It acts like a python dictionary where the keys are field names and returns 3D numpy arrays of whatever uniform resolution you specify when you create the covering_grid.
Note that if you're using load_particles all of your data needs to live in memory. If your data is too big for that you'll need to write a frontend for your data format or use a memmap to an on-disk file somehow. I'm not an expert on that but others on the list should be able to help out.
Hope that gets you well on your way :)
-Nathan
On Thu, Jun 5, 2014 at 5:04 PM, Desika Narayanan <dnarayan@haverford.edu> wrote:
Hey Brendan,
A couple of extra tools you might find helpful in conjunction with Nathan's example of depositing the particles onto an octree are at:
http://paste.yt-project.org/show/4737/
Where I load a gadget snapshot, and then recover the coordinates and width of each cell.
In response to your last question - the particles are deposited into an octree grid (so, you'll see that the cell sizes aren't all the same size). I don't know if depositing onto a regular NxNxN mesh is possible, though would be interested to hear if so.
-d
On Thu, Jun 5, 2014 at 7:58 PM, Brendan Griffen < brendan.f.griffen@gmail.com> wrote:
Thanks. I'll get the "bleeding edge" version first then try your suggestions. Though I want to return the NxNxN array and be able to write this mesh to a file. It is *only* using the cic part of yt and it should return the mesh to be written? Just wanted to clarify?
Thanks. Brendan
On Thu, Jun 5, 2014 at 6:49 PM, Nathan Goldbaum <nathan12343@gmail.com> wrote:
On Thu, Jun 5, 2014 at 3:36 PM, John ZuHone <jzuhone@gmail.com> wrote:
Hi Brendan,
Which version of yt are you using?
If you're using 3.0, this is actually fairly easy. If you look in yt.fields.particle_fields.py, around line 85, you can see how this is done for the "particle_density" and "particle_mass" fields. Basically you can call a "deposit" method which takes the particle field quantity you want deposited and deposits it into cells. The underlying calculation is done using Cython, so it's fast.
And you shouldn't ever actually need to call these "deposit" functions, since "deposit" is exposed as a field type for all datasets that contain particles.
Here is a notebook that does this for Enzo AMR data:
http://nbviewer.ipython.org/gist/ngoldbaum/5e19e4e6cc2bf330149c
This dataset contains about a million particles and generates a CIC deposition for the whole domain in about 6 seconds from a cold start.
If you're using 2.x, then you can do the same thing, but it's not as straightforward. You can see how this works in yt.data_objects.universal_fields.py, around line 986, where the "particle_density" field is defined. Basically, it calls CICDeposit_3, which is in yt.utilities.lib.CICDeposit.pyx.
Let me know if you need any more clarification.
Best,
John Z
On Jun 5, 2014, at 6:07 PM, Brendan Griffen < brendan.f.griffen@gmail.com> wrote:
Hi,
I was wondering if there were any Cython routines within yt which takes particle data and converts it into a cloud-in-cell based mesh which can be written to a file of my choosing.
What sort of mesh were you looking for? yt will internally construct an octree if it is fed particle data. I'm not sure whether this octree can be saved to disk for later analysis.
It's also possible to create a uniform resolution covering grid containing field data for a deposited quantity, which can be quite easily saved to disk in a number of ways.
I heard a while ago there was some such functionality but it could be too far down the yt rabbit hole to be used as a standalone? Is this true? I have my own Python code for doing it but it just isn't fast enough and thought I'd ask the yt community if there were any wrapper tools available to boost the speed.
Thanks. Brendan _______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
That's right, you can set that via the bbox keyword parameter for load_particles. I'd urge you to take a look at the docstrings and source code for load_particles. On Thu, Jun 5, 2014 at 6:34 PM, Brendan Griffen <brendan.f.griffen@gmail.com
wrote:
Thanks very much Nathan. I tried to load in my own data but I think there are too many particles or I have to specifically set the domain size.
In this area:
data = {'particle_position_x': pos[:,0], 'particle_position_y': pos[:,1], 'particle_position_z': pos[:,2], 'particle_mass': np.array([mpart]*npart)}
ds = yt.load_particles(data, length_unit=3.08e24, mass_unit=1.9891e36) ad = ds.all_data() print ad['deposit', 'all_cic']
In [3]: run ytcic.py yt : [INFO ] 2014-06-05 21:29:06,183 Parameters: current_time = 0.0 yt : [INFO ] 2014-06-05 21:29:06,183 Parameters: domain_dimensions = [2 2 2] yt : [INFO ] 2014-06-05 21:29:06,184 Parameters: domain_left_edge = [ 0. 0. 0.] yt : [INFO ] 2014-06-05 21:29:06,185 Parameters: domain_right_edge = [ 1. 1. 1.] yt : [INFO ] 2014-06-05 21:29:06,185 Parameters: cosmological_simulation = 0.0 yt : [INFO ] 2014-06-05 21:29:06,188 Allocating for 1.342e+08 particles --------------------------------------------------------------------------- YTDomainOverflow Traceback (most recent call last) /nfs/blank/h4231/bgriffen/data/lib/yt-x86_64/lib/python2.7/site-packages/IPython/utils/py3compat.pyc in execfile(fname, *where) 202 else: 203 filename = fname --> 204 __builtin__.execfile(filename, *where)
/nfs/blank/h4231/bgriffen/work/projects/caterpillar/c2ray/cic/ytcic.py in <module>() 52 53 ad = ds.all_data() ---> 54 print ad['deposit', 'all_cic'] 55 slc = yt.SlicePlot(ds, 2, ('deposit', 'all_cic')) 56 slc.set_figure_size(4)
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc in __getitem__(self, key) 205 Returns a single field. Will add if necessary. 206 """ --> 207 f = self._determine_fields([key])[0] 208 if f not in self.field_data and key not in self.field_data: 209 if f in self._container_fields:
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc in _determine_fields(self, fields) 453 raise YTFieldNotParseable(field) 454 ftype, fname = field --> 455 finfo = self.pf._get_field_info(ftype, fname) 456 else: 457 fname = field
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/static_output.pyc in _get_field_info(self, ftype, fname) 445 _last_finfo = None 446 def _get_field_info(self, ftype, fname = None): --> 447 self.index 448 if fname is None: 449 ftype, fname = "unknown", ftype
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/static_output.pyc in index(self) 277 raise RuntimeError("You should not instantiate Dataset.") 278 self._instantiated_index = self._index_class( --> 279 self, dataset_type=self.dataset_type) 280 # Now we do things that we need an instantiated index for 281 # ...first off, we create our field_info now.
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/frontends/stream/data_structures.pyc in __init__(self, pf, dataset_type) 942 def __init__(self, pf, dataset_type = None): 943 self.stream_handler = pf.stream_handler --> 944 super(StreamParticleIndex, self).__init__(pf, dataset_type) 945 946 def _setup_data_io(self):
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/geometry/particle_geometry_handler.pyc in __init__(self, pf, dataset_type) 48 self.directory = os.path.dirname(self.index_filename) 49 self.float_type = np.float64 ---> 50 super(ParticleIndex, self).__init__(pf, dataset_type) 51 52 def _setup_geometry(self):
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/geometry/geometry_handler.pyc in __init__(self, pf, dataset_type) 54 55 mylog.debug("Setting up domain geometry.") ---> 56 self._setup_geometry() 57 58 mylog.debug("Initializing data grid data IO")
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/geometry/particle_geometry_handler.pyc in _setup_geometry(self) 52 def _setup_geometry(self): 53 mylog.debug("Initializing Particle Geometry Handler.") ---> 54 self._initialize_particle_handler() 55 56
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/geometry/particle_geometry_handler.pyc in _initialize_particle_handler(self) 87 pf.domain_left_edge, pf.domain_right_edge, 88 [N, N, N], len(self.data_files)) ---> 89 self._initialize_indices() 90 self.oct_handler.finalize() 91 self.max_level = self.oct_handler.max_level
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/geometry/particle_geometry_handler.pyc in _initialize_indices(self) 109 npart = sum(data_file.total_particles.values()) 110 morton[ind:ind + npart] = \ --> 111 self.io._initialize_index(data_file, self.regions) 112 ind += npart 113 morton.sort()
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/frontends/stream/io.pyc in _initialize_index(self, data_file, regions) 144 raise YTDomainOverflow(pos.min(axis=0), pos.max(axis=0), 145 data_file.pf.domain_left_edge, --> 146 data_file.pf.domain_right_edge) 147 regions.add_data_file(pos, data_file.file_id) 148 morton.append(compute_morton(
YTDomainOverflow: Particle bounds [ 0. 0. 0.] and [ 99.99999237 99.99999237 99.99999237] exceed domain bounds [ 0. 0. 0.] code_length and [ 1. 1. 1.] code_length
On Thu, Jun 5, 2014 at 8:22 PM, Nathan Goldbaum <nathan12343@gmail.com> wrote:
Here's a worked out example that does what you're looking for using a fake 1 million particle dataset:
http://nbviewer.ipython.org/gist/ngoldbaum/546d37869aafe71cfe38
In this notebook I make use of two key yt features: `load_particles`, and `covering_grid`.
load_particles creates a "stream" dataset based on in-memory data fed in as a numpy array. This dataset acts just like an on-disk simulation dataset, but doesn't come with the baggage of needing to write a custom frontend to read a specific data format off disk.
covering_grid is a way to generate uniform resolution data from an AMR dataset. It acts like a python dictionary where the keys are field names and returns 3D numpy arrays of whatever uniform resolution you specify when you create the covering_grid.
Note that if you're using load_particles all of your data needs to live in memory. If your data is too big for that you'll need to write a frontend for your data format or use a memmap to an on-disk file somehow. I'm not an expert on that but others on the list should be able to help out.
Hope that gets you well on your way :)
-Nathan
On Thu, Jun 5, 2014 at 5:04 PM, Desika Narayanan <dnarayan@haverford.edu> wrote:
Hey Brendan,
A couple of extra tools you might find helpful in conjunction with Nathan's example of depositing the particles onto an octree are at:
http://paste.yt-project.org/show/4737/
Where I load a gadget snapshot, and then recover the coordinates and width of each cell.
In response to your last question - the particles are deposited into an octree grid (so, you'll see that the cell sizes aren't all the same size). I don't know if depositing onto a regular NxNxN mesh is possible, though would be interested to hear if so.
-d
On Thu, Jun 5, 2014 at 7:58 PM, Brendan Griffen < brendan.f.griffen@gmail.com> wrote:
Thanks. I'll get the "bleeding edge" version first then try your suggestions. Though I want to return the NxNxN array and be able to write this mesh to a file. It is *only* using the cic part of yt and it should return the mesh to be written? Just wanted to clarify?
Thanks. Brendan
On Thu, Jun 5, 2014 at 6:49 PM, Nathan Goldbaum <nathan12343@gmail.com> wrote:
On Thu, Jun 5, 2014 at 3:36 PM, John ZuHone <jzuhone@gmail.com> wrote:
Hi Brendan,
Which version of yt are you using?
If you're using 3.0, this is actually fairly easy. If you look in yt.fields.particle_fields.py, around line 85, you can see how this is done for the "particle_density" and "particle_mass" fields. Basically you can call a "deposit" method which takes the particle field quantity you want deposited and deposits it into cells. The underlying calculation is done using Cython, so it's fast.
And you shouldn't ever actually need to call these "deposit" functions, since "deposit" is exposed as a field type for all datasets that contain particles.
Here is a notebook that does this for Enzo AMR data:
http://nbviewer.ipython.org/gist/ngoldbaum/5e19e4e6cc2bf330149c
This dataset contains about a million particles and generates a CIC deposition for the whole domain in about 6 seconds from a cold start.
If you're using 2.x, then you can do the same thing, but it's not as straightforward. You can see how this works in yt.data_objects.universal_fields.py, around line 986, where the "particle_density" field is defined. Basically, it calls CICDeposit_3, which is in yt.utilities.lib.CICDeposit.pyx.
Let me know if you need any more clarification.
Best,
John Z
On Jun 5, 2014, at 6:07 PM, Brendan Griffen < brendan.f.griffen@gmail.com> wrote:
> Hi, > > I was wondering if there were any Cython routines within yt which takes particle data and converts it into a cloud-in-cell based mesh which can be written to a file of my choosing.
What sort of mesh were you looking for? yt will internally construct an octree if it is fed particle data. I'm not sure whether this octree can be saved to disk for later analysis.
It's also possible to create a uniform resolution covering grid containing field data for a deposited quantity, which can be quite easily saved to disk in a number of ways.
I heard a while ago there was some such functionality but it could be too far down the yt rabbit hole to be used as a standalone? Is this true? I have my own Python code for doing it but it just isn't fast enough and thought I'd ask the yt community if there were any wrapper tools available to boost the speed. > > Thanks. > Brendan > _______________________________________________ > yt-users mailing list > yt-users@lists.spacepope.org > http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
OK great. Thanks. I just wanted a homogeneous mesh. 512^3 with no nesting of any kind. Though when I plot the image it looks like it is assigning particles incorrectly (low resolution on the outside). This is just a test image. ds = yt.load_particles(data, length_unit=3.08e24, mass_unit=1.9891e33,bbox=bbox) ad = ds.all_data() print ad['deposit', 'all_cic'] slc = yt.SlicePlot(ds, 2, ('deposit', 'all_cic')) slc.set_figure_size(4) cg = ds.covering_grid(level=9, left_edge=[0,0,0],dims=[512,512,512]) Also, is there a way to load multiple particle types? Do I just need to stack the particles into the array here? data = {'particle_position_x': pos[:,0], 'particle_position_y': pos[:,1], 'particle_position_z': pos[:,2], 'particle_mass': np.array([mpart]*npart)} Then feed it in as usual? [image: Inline image 1] Brendan On Thu, Jun 5, 2014 at 9:44 PM, Nathan Goldbaum <nathan12343@gmail.com> wrote:
That's right, you can set that via the bbox keyword parameter for load_particles. I'd urge you to take a look at the docstrings and source code for load_particles.
On Thu, Jun 5, 2014 at 6:34 PM, Brendan Griffen < brendan.f.griffen@gmail.com> wrote:
Thanks very much Nathan. I tried to load in my own data but I think there are too many particles or I have to specifically set the domain size.
In this area:
data = {'particle_position_x': pos[:,0], 'particle_position_y': pos[:,1], 'particle_position_z': pos[:,2], 'particle_mass': np.array([mpart]*npart)}
ds = yt.load_particles(data, length_unit=3.08e24, mass_unit=1.9891e36) ad = ds.all_data() print ad['deposit', 'all_cic']
In [3]: run ytcic.py yt : [INFO ] 2014-06-05 21:29:06,183 Parameters: current_time = 0.0 yt : [INFO ] 2014-06-05 21:29:06,183 Parameters: domain_dimensions = [2 2 2] yt : [INFO ] 2014-06-05 21:29:06,184 Parameters: domain_left_edge = [ 0. 0. 0.] yt : [INFO ] 2014-06-05 21:29:06,185 Parameters: domain_right_edge = [ 1. 1. 1.] yt : [INFO ] 2014-06-05 21:29:06,185 Parameters: cosmological_simulation = 0.0 yt : [INFO ] 2014-06-05 21:29:06,188 Allocating for 1.342e+08 particles
--------------------------------------------------------------------------- YTDomainOverflow Traceback (most recent call last) /nfs/blank/h4231/bgriffen/data/lib/yt-x86_64/lib/python2.7/site-packages/IPython/utils/py3compat.pyc in execfile(fname, *where) 202 else: 203 filename = fname --> 204 __builtin__.execfile(filename, *where)
/nfs/blank/h4231/bgriffen/work/projects/caterpillar/c2ray/cic/ytcic.py in <module>() 52 53 ad = ds.all_data() ---> 54 print ad['deposit', 'all_cic'] 55 slc = yt.SlicePlot(ds, 2, ('deposit', 'all_cic')) 56 slc.set_figure_size(4)
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc in __getitem__(self, key) 205 Returns a single field. Will add if necessary. 206 """ --> 207 f = self._determine_fields([key])[0] 208 if f not in self.field_data and key not in self.field_data: 209 if f in self._container_fields:
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc in _determine_fields(self, fields) 453 raise YTFieldNotParseable(field) 454 ftype, fname = field --> 455 finfo = self.pf._get_field_info(ftype, fname) 456 else: 457 fname = field
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/static_output.pyc in _get_field_info(self, ftype, fname) 445 _last_finfo = None 446 def _get_field_info(self, ftype, fname = None): --> 447 self.index 448 if fname is None: 449 ftype, fname = "unknown", ftype
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/static_output.pyc in index(self) 277 raise RuntimeError("You should not instantiate Dataset.") 278 self._instantiated_index = self._index_class( --> 279 self, dataset_type=self.dataset_type) 280 # Now we do things that we need an instantiated index for 281 # ...first off, we create our field_info now.
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/frontends/stream/data_structures.pyc in __init__(self, pf, dataset_type) 942 def __init__(self, pf, dataset_type = None): 943 self.stream_handler = pf.stream_handler --> 944 super(StreamParticleIndex, self).__init__(pf, dataset_type) 945 946 def _setup_data_io(self):
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/geometry/particle_geometry_handler.pyc in __init__(self, pf, dataset_type) 48 self.directory = os.path.dirname(self.index_filename) 49 self.float_type = np.float64 ---> 50 super(ParticleIndex, self).__init__(pf, dataset_type) 51 52 def _setup_geometry(self):
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/geometry/geometry_handler.pyc in __init__(self, pf, dataset_type) 54 55 mylog.debug("Setting up domain geometry.") ---> 56 self._setup_geometry() 57 58 mylog.debug("Initializing data grid data IO")
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/geometry/particle_geometry_handler.pyc in _setup_geometry(self) 52 def _setup_geometry(self): 53 mylog.debug("Initializing Particle Geometry Handler.") ---> 54 self._initialize_particle_handler() 55 56
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/geometry/particle_geometry_handler.pyc in _initialize_particle_handler(self) 87 pf.domain_left_edge, pf.domain_right_edge, 88 [N, N, N], len(self.data_files)) ---> 89 self._initialize_indices() 90 self.oct_handler.finalize() 91 self.max_level = self.oct_handler.max_level
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/geometry/particle_geometry_handler.pyc in _initialize_indices(self) 109 npart = sum(data_file.total_particles.values()) 110 morton[ind:ind + npart] = \ --> 111 self.io._initialize_index(data_file, self.regions) 112 ind += npart 113 morton.sort()
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/frontends/stream/io.pyc in _initialize_index(self, data_file, regions) 144 raise YTDomainOverflow(pos.min(axis=0), pos.max(axis=0), 145 data_file.pf.domain_left_edge, --> 146 data_file.pf.domain_right_edge) 147 regions.add_data_file(pos, data_file.file_id) 148 morton.append(compute_morton(
YTDomainOverflow: Particle bounds [ 0. 0. 0.] and [ 99.99999237 99.99999237 99.99999237] exceed domain bounds [ 0. 0. 0.] code_length and [ 1. 1. 1.] code_length
On Thu, Jun 5, 2014 at 8:22 PM, Nathan Goldbaum <nathan12343@gmail.com> wrote:
Here's a worked out example that does what you're looking for using a fake 1 million particle dataset:
http://nbviewer.ipython.org/gist/ngoldbaum/546d37869aafe71cfe38
In this notebook I make use of two key yt features: `load_particles`, and `covering_grid`.
load_particles creates a "stream" dataset based on in-memory data fed in as a numpy array. This dataset acts just like an on-disk simulation dataset, but doesn't come with the baggage of needing to write a custom frontend to read a specific data format off disk.
covering_grid is a way to generate uniform resolution data from an AMR dataset. It acts like a python dictionary where the keys are field names and returns 3D numpy arrays of whatever uniform resolution you specify when you create the covering_grid.
Note that if you're using load_particles all of your data needs to live in memory. If your data is too big for that you'll need to write a frontend for your data format or use a memmap to an on-disk file somehow. I'm not an expert on that but others on the list should be able to help out.
Hope that gets you well on your way :)
-Nathan
On Thu, Jun 5, 2014 at 5:04 PM, Desika Narayanan <dnarayan@haverford.edu
wrote:
Hey Brendan,
A couple of extra tools you might find helpful in conjunction with Nathan's example of depositing the particles onto an octree are at:
http://paste.yt-project.org/show/4737/
Where I load a gadget snapshot, and then recover the coordinates and width of each cell.
In response to your last question - the particles are deposited into an octree grid (so, you'll see that the cell sizes aren't all the same size). I don't know if depositing onto a regular NxNxN mesh is possible, though would be interested to hear if so.
-d
On Thu, Jun 5, 2014 at 7:58 PM, Brendan Griffen < brendan.f.griffen@gmail.com> wrote:
Thanks. I'll get the "bleeding edge" version first then try your suggestions. Though I want to return the NxNxN array and be able to write this mesh to a file. It is *only* using the cic part of yt and it should return the mesh to be written? Just wanted to clarify?
Thanks. Brendan
On Thu, Jun 5, 2014 at 6:49 PM, Nathan Goldbaum <nathan12343@gmail.com
wrote:
On Thu, Jun 5, 2014 at 3:36 PM, John ZuHone <jzuhone@gmail.com> wrote:
> Hi Brendan, > > Which version of yt are you using? > > If you're using 3.0, this is actually fairly easy. If you look in > yt.fields.particle_fields.py, around line 85, you can see how this > is done for the "particle_density" and "particle_mass" fields. Basically > you can call a "deposit" method which takes the particle field quantity you > want deposited and deposits it into cells. The underlying calculation is > done using Cython, so it's fast. >
And you shouldn't ever actually need to call these "deposit" functions, since "deposit" is exposed as a field type for all datasets that contain particles.
Here is a notebook that does this for Enzo AMR data:
http://nbviewer.ipython.org/gist/ngoldbaum/5e19e4e6cc2bf330149c
This dataset contains about a million particles and generates a CIC deposition for the whole domain in about 6 seconds from a cold start.
> > If you're using 2.x, then you can do the same thing, but it's not as > straightforward. You can see how this works in > yt.data_objects.universal_fields.py, around line 986, where the > "particle_density" field is defined. Basically, it calls CICDeposit_3, > which is in yt.utilities.lib.CICDeposit.pyx. > > Let me know if you need any more clarification. > > Best, > > John Z > > On Jun 5, 2014, at 6:07 PM, Brendan Griffen < > brendan.f.griffen@gmail.com> wrote: > > > Hi, > > > > I was wondering if there were any Cython routines within yt which > takes particle data and converts it into a cloud-in-cell based mesh which > can be written to a file of my choosing. >
What sort of mesh were you looking for? yt will internally construct an octree if it is fed particle data. I'm not sure whether this octree can be saved to disk for later analysis.
It's also possible to create a uniform resolution covering grid containing field data for a deposited quantity, which can be quite easily saved to disk in a number of ways.
> I heard a while ago there was some such functionality but it could > be too far down the yt rabbit hole to be used as a standalone? Is this > true? I have my own Python code for doing it but it just isn't fast enough > and thought I'd ask the yt community if there were any wrapper tools > available to boost the speed. > > > > Thanks. > > Brendan > > _______________________________________________ > > yt-users mailing list > > yt-users@lists.spacepope.org > > http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org > > _______________________________________________ > yt-users mailing list > yt-users@lists.spacepope.org > http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
On Fri, Jun 6, 2014 at 11:48 AM, Brendan Griffen < brendan.f.griffen@gmail.com> wrote:
OK great. Thanks. I just wanted a homogeneous mesh. 512^3 with no nesting of any kind. Though when I plot the image it looks like it is assigning particles incorrectly (low resolution on the outside). This is just a test image.
The SlicePlot is visualizing the octree so there is less resolution where there are fewer particles. If you want to visualize the covering grid you're going to need to visualize that separately.
ds = yt.load_particles(data, length_unit=3.08e24, mass_unit=1.9891e33,bbox=bbox)
ad = ds.all_data() print ad['deposit', 'all_cic'] slc = yt.SlicePlot(ds, 2, ('deposit', 'all_cic')) slc.set_figure_size(4) cg = ds.covering_grid(level=9, left_edge=[0,0,0],dims=[512,512,512])
To actually produce the uniform resolution ndarray, you're going to need to do something like: array = cg[('deposit', 'all_cic')] array will then be a 3D array you can do whatever you want with. By default it has units, but to strip them off you'll just need to cast to ndarray: array_without_units = array.v
Also, is there a way to load multiple particle types?
Do I just need to stack the particles into the array here?
data = {'particle_position_x': pos[:,0], 'particle_position_y': pos[:,1], 'particle_position_z': pos[:,2], 'particle_mass': np.array([mpart]*npart)}
Then feed it in as usual?
That's right, although if the particle masses are different for the different particle types that code snippet will need to be generalized to handle that. I think in principle it should be possible to make load_particles handle different particle types just like an SPH dataset that contains multiple particle types, but right now that hasn't been implemented yet.
[image: Inline image 1] Brendan
On Thu, Jun 5, 2014 at 9:44 PM, Nathan Goldbaum <nathan12343@gmail.com> wrote:
That's right, you can set that via the bbox keyword parameter for load_particles. I'd urge you to take a look at the docstrings and source code for load_particles.
On Thu, Jun 5, 2014 at 6:34 PM, Brendan Griffen < brendan.f.griffen@gmail.com> wrote:
Thanks very much Nathan. I tried to load in my own data but I think there are too many particles or I have to specifically set the domain size.
In this area:
data = {'particle_position_x': pos[:,0], 'particle_position_y': pos[:,1], 'particle_position_z': pos[:,2], 'particle_mass': np.array([mpart]*npart)}
ds = yt.load_particles(data, length_unit=3.08e24, mass_unit=1.9891e36) ad = ds.all_data() print ad['deposit', 'all_cic']
In [3]: run ytcic.py yt : [INFO ] 2014-06-05 21:29:06,183 Parameters: current_time = 0.0 yt : [INFO ] 2014-06-05 21:29:06,183 Parameters: domain_dimensions = [2 2 2] yt : [INFO ] 2014-06-05 21:29:06,184 Parameters: domain_left_edge = [ 0. 0. 0.] yt : [INFO ] 2014-06-05 21:29:06,185 Parameters: domain_right_edge = [ 1. 1. 1.] yt : [INFO ] 2014-06-05 21:29:06,185 Parameters: cosmological_simulation = 0.0 yt : [INFO ] 2014-06-05 21:29:06,188 Allocating for 1.342e+08 particles
--------------------------------------------------------------------------- YTDomainOverflow Traceback (most recent call last) /nfs/blank/h4231/bgriffen/data/lib/yt-x86_64/lib/python2.7/site-packages/IPython/utils/py3compat.pyc in execfile(fname, *where) 202 else: 203 filename = fname --> 204 __builtin__.execfile(filename, *where)
/nfs/blank/h4231/bgriffen/work/projects/caterpillar/c2ray/cic/ytcic.py in <module>() 52 53 ad = ds.all_data() ---> 54 print ad['deposit', 'all_cic'] 55 slc = yt.SlicePlot(ds, 2, ('deposit', 'all_cic')) 56 slc.set_figure_size(4)
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc in __getitem__(self, key) 205 Returns a single field. Will add if necessary. 206 """ --> 207 f = self._determine_fields([key])[0] 208 if f not in self.field_data and key not in self.field_data: 209 if f in self._container_fields:
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc in _determine_fields(self, fields) 453 raise YTFieldNotParseable(field) 454 ftype, fname = field --> 455 finfo = self.pf._get_field_info(ftype, fname) 456 else: 457 fname = field
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/static_output.pyc in _get_field_info(self, ftype, fname) 445 _last_finfo = None 446 def _get_field_info(self, ftype, fname = None): --> 447 self.index 448 if fname is None: 449 ftype, fname = "unknown", ftype
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/static_output.pyc in index(self) 277 raise RuntimeError("You should not instantiate Dataset.") 278 self._instantiated_index = self._index_class( --> 279 self, dataset_type=self.dataset_type) 280 # Now we do things that we need an instantiated index for 281 # ...first off, we create our field_info now.
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/frontends/stream/data_structures.pyc in __init__(self, pf, dataset_type) 942 def __init__(self, pf, dataset_type = None): 943 self.stream_handler = pf.stream_handler --> 944 super(StreamParticleIndex, self).__init__(pf, dataset_type) 945 946 def _setup_data_io(self):
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/geometry/particle_geometry_handler.pyc in __init__(self, pf, dataset_type) 48 self.directory = os.path.dirname(self.index_filename) 49 self.float_type = np.float64 ---> 50 super(ParticleIndex, self).__init__(pf, dataset_type) 51 52 def _setup_geometry(self):
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/geometry/geometry_handler.pyc in __init__(self, pf, dataset_type) 54 55 mylog.debug("Setting up domain geometry.") ---> 56 self._setup_geometry() 57 58 mylog.debug("Initializing data grid data IO")
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/geometry/particle_geometry_handler.pyc in _setup_geometry(self) 52 def _setup_geometry(self): 53 mylog.debug("Initializing Particle Geometry Handler.") ---> 54 self._initialize_particle_handler() 55 56
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/geometry/particle_geometry_handler.pyc in _initialize_particle_handler(self) 87 pf.domain_left_edge, pf.domain_right_edge, 88 [N, N, N], len(self.data_files)) ---> 89 self._initialize_indices() 90 self.oct_handler.finalize() 91 self.max_level = self.oct_handler.max_level
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/geometry/particle_geometry_handler.pyc in _initialize_indices(self) 109 npart = sum(data_file.total_particles.values()) 110 morton[ind:ind + npart] = \ --> 111 self.io._initialize_index(data_file, self.regions) 112 ind += npart 113 morton.sort()
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/frontends/stream/io.pyc in _initialize_index(self, data_file, regions) 144 raise YTDomainOverflow(pos.min(axis=0), pos.max(axis=0), 145 data_file.pf.domain_left_edge, --> 146 data_file.pf.domain_right_edge) 147 regions.add_data_file(pos, data_file.file_id) 148 morton.append(compute_morton(
YTDomainOverflow: Particle bounds [ 0. 0. 0.] and [ 99.99999237 99.99999237 99.99999237] exceed domain bounds [ 0. 0. 0.] code_length and [ 1. 1. 1.] code_length
On Thu, Jun 5, 2014 at 8:22 PM, Nathan Goldbaum <nathan12343@gmail.com> wrote:
Here's a worked out example that does what you're looking for using a fake 1 million particle dataset:
http://nbviewer.ipython.org/gist/ngoldbaum/546d37869aafe71cfe38
In this notebook I make use of two key yt features: `load_particles`, and `covering_grid`.
load_particles creates a "stream" dataset based on in-memory data fed in as a numpy array. This dataset acts just like an on-disk simulation dataset, but doesn't come with the baggage of needing to write a custom frontend to read a specific data format off disk.
covering_grid is a way to generate uniform resolution data from an AMR dataset. It acts like a python dictionary where the keys are field names and returns 3D numpy arrays of whatever uniform resolution you specify when you create the covering_grid.
Note that if you're using load_particles all of your data needs to live in memory. If your data is too big for that you'll need to write a frontend for your data format or use a memmap to an on-disk file somehow. I'm not an expert on that but others on the list should be able to help out.
Hope that gets you well on your way :)
-Nathan
On Thu, Jun 5, 2014 at 5:04 PM, Desika Narayanan < dnarayan@haverford.edu> wrote:
Hey Brendan,
A couple of extra tools you might find helpful in conjunction with Nathan's example of depositing the particles onto an octree are at:
http://paste.yt-project.org/show/4737/
Where I load a gadget snapshot, and then recover the coordinates and width of each cell.
In response to your last question - the particles are deposited into an octree grid (so, you'll see that the cell sizes aren't all the same size). I don't know if depositing onto a regular NxNxN mesh is possible, though would be interested to hear if so.
-d
On Thu, Jun 5, 2014 at 7:58 PM, Brendan Griffen < brendan.f.griffen@gmail.com> wrote:
Thanks. I'll get the "bleeding edge" version first then try your suggestions. Though I want to return the NxNxN array and be able to write this mesh to a file. It is *only* using the cic part of yt and it should return the mesh to be written? Just wanted to clarify?
Thanks. Brendan
On Thu, Jun 5, 2014 at 6:49 PM, Nathan Goldbaum < nathan12343@gmail.com> wrote:
> > > > On Thu, Jun 5, 2014 at 3:36 PM, John ZuHone <jzuhone@gmail.com> > wrote: > >> Hi Brendan, >> >> Which version of yt are you using? >> >> If you're using 3.0, this is actually fairly easy. If you look in >> yt.fields.particle_fields.py, around line 85, you can see how this >> is done for the "particle_density" and "particle_mass" fields. Basically >> you can call a "deposit" method which takes the particle field quantity you >> want deposited and deposits it into cells. The underlying calculation is >> done using Cython, so it's fast. >> > > And you shouldn't ever actually need to call these "deposit" > functions, since "deposit" is exposed as a field type for all datasets that > contain particles. > > Here is a notebook that does this for Enzo AMR data: > > http://nbviewer.ipython.org/gist/ngoldbaum/5e19e4e6cc2bf330149c > > This dataset contains about a million particles and generates a CIC > deposition for the whole domain in about 6 seconds from a cold start. > > >> >> If you're using 2.x, then you can do the same thing, but it's not >> as straightforward. You can see how this works in >> yt.data_objects.universal_fields.py, around line 986, where the >> "particle_density" field is defined. Basically, it calls CICDeposit_3, >> which is in yt.utilities.lib.CICDeposit.pyx. >> >> Let me know if you need any more clarification. >> >> Best, >> >> John Z >> >> On Jun 5, 2014, at 6:07 PM, Brendan Griffen < >> brendan.f.griffen@gmail.com> wrote: >> >> > Hi, >> > >> > I was wondering if there were any Cython routines within yt which >> takes particle data and converts it into a cloud-in-cell based mesh which >> can be written to a file of my choosing. >> > > What sort of mesh were you looking for? yt will internally > construct an octree if it is fed particle data. I'm not sure whether this > octree can be saved to disk for later analysis. > > It's also possible to create a uniform resolution covering grid > containing field data for a deposited quantity, which can be quite easily > saved to disk in a number of ways. > > >> I heard a while ago there was some such functionality but it could >> be too far down the yt rabbit hole to be used as a standalone? Is this >> true? I have my own Python code for doing it but it just isn't fast enough >> and thought I'd ask the yt community if there were any wrapper tools >> available to boost the speed. >> > >> > Thanks. >> > Brendan >> > _______________________________________________ >> > yt-users mailing list >> > yt-users@lists.spacepope.org >> > http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >> >> _______________________________________________ >> yt-users mailing list >> yt-users@lists.spacepope.org >> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >> > > > _______________________________________________ > yt-users mailing list > yt-users@lists.spacepope.org > http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org > >
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
OK great. It is very low resolution but it worked. Thanks for all your help. My higher resolution run 1024^3 in 100 Mpc seems to crash on 128GB memory machine. I might have to look elsewhere. Also, I normally use Canopy distribution but I just use an alias to loadyt which erases my PYTHONPATH and I can't access scipy and a few other libraries any more. What is the best practice here? Should I just manually export PYTHONPATH and point to the libraries need in canopy or can they play nice together? Thanks. BG .[image: Inline image 1]
On Fri, Jun 6, 2014 at 1:16 PM, Brendan Griffen <brendan.f.griffen@gmail.com
wrote:
OK great. It is very low resolution but it worked. Thanks for all your help. My higher resolution run 1024^3 in 100 Mpc seems to crash on 128GB memory machine. I might have to look elsewhere.
Hmm, for a 1024^3 dataset it might be best to read the data off-disk instead of using load_particles. We're getting into territory that I'm entirely unfamiliar with, although others on here might be able to help dealing with very large datasets.
Also, I normally use Canopy distribution but I just use an alias to loadyt which erases my PYTHONPATH and I can't access scipy and a few other libraries any more. What is the best practice here? Should I just manually export PYTHONPATH and point to the libraries need in canopy or can they play nice together?
Rather than using the install script, you can build yt inside of canopy's python environment by doing: cd /directory/to/put/yt hg clone https://bitbucket.org/yt_analysis/yt ./yt-hg cd yt-hg python setup.py develop With yt-3.0 and canppy this should work right out of the box.
Thanks.
BG
.[image: Inline image 1]
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
OK great. It is very low resolution but it worked. Thanks for all your help. My higher resolution run 1024^3 in 100 Mpc seems to crash on 128GB memory machine. I might have to look elsewhere. Also, I normally use Canopy distribution but I just use an alias to loadyt which erases my PYTHONPATH and I can't access scipy and a few other libraries any more. What is the best practice here? Should I just manually export PYTHONPATH and point to the libraries need in canopy or can they play nice together? Thanks. BG On Fri, Jun 6, 2014 at 2:54 PM, Nathan Goldbaum <nathan12343@gmail.com> wrote:
On Fri, Jun 6, 2014 at 11:48 AM, Brendan Griffen < brendan.f.griffen@gmail.com> wrote:
OK great. Thanks. I just wanted a homogeneous mesh. 512^3 with no nesting of any kind. Though when I plot the image it looks like it is assigning particles incorrectly (low resolution on the outside). This is just a test image.
The SlicePlot is visualizing the octree so there is less resolution where there are fewer particles. If you want to visualize the covering grid you're going to need to visualize that separately.
ds = yt.load_particles(data, length_unit=3.08e24, mass_unit=1.9891e33,bbox=bbox)
ad = ds.all_data() print ad['deposit', 'all_cic'] slc = yt.SlicePlot(ds, 2, ('deposit', 'all_cic')) slc.set_figure_size(4) cg = ds.covering_grid(level=9, left_edge=[0,0,0],dims=[512,512,512])
To actually produce the uniform resolution ndarray, you're going to need to do something like:
array = cg[('deposit', 'all_cic')]
array will then be a 3D array you can do whatever you want with. By default it has units, but to strip them off you'll just need to cast to ndarray:
array_without_units = array.v
Also, is there a way to load multiple particle types?
Do I just need to stack the particles into the array here?
data = {'particle_position_x': pos[:,0], 'particle_position_y': pos[:,1], 'particle_position_z': pos[:,2], 'particle_mass': np.array([mpart]*npart)}
Then feed it in as usual?
That's right, although if the particle masses are different for the different particle types that code snippet will need to be generalized to handle that.
I think in principle it should be possible to make load_particles handle different particle types just like an SPH dataset that contains multiple particle types, but right now that hasn't been implemented yet.
[image: Inline image 1] Brendan
On Thu, Jun 5, 2014 at 9:44 PM, Nathan Goldbaum <nathan12343@gmail.com> wrote:
That's right, you can set that via the bbox keyword parameter for load_particles. I'd urge you to take a look at the docstrings and source code for load_particles.
On Thu, Jun 5, 2014 at 6:34 PM, Brendan Griffen < brendan.f.griffen@gmail.com> wrote:
Thanks very much Nathan. I tried to load in my own data but I think there are too many particles or I have to specifically set the domain size.
In this area:
data = {'particle_position_x': pos[:,0], 'particle_position_y': pos[:,1], 'particle_position_z': pos[:,2], 'particle_mass': np.array([mpart]*npart)}
ds = yt.load_particles(data, length_unit=3.08e24, mass_unit=1.9891e36) ad = ds.all_data() print ad['deposit', 'all_cic']
In [3]: run ytcic.py yt : [INFO ] 2014-06-05 21:29:06,183 Parameters: current_time = 0.0 yt : [INFO ] 2014-06-05 21:29:06,183 Parameters: domain_dimensions = [2 2 2] yt : [INFO ] 2014-06-05 21:29:06,184 Parameters: domain_left_edge = [ 0. 0. 0.] yt : [INFO ] 2014-06-05 21:29:06,185 Parameters: domain_right_edge = [ 1. 1. 1.] yt : [INFO ] 2014-06-05 21:29:06,185 Parameters: cosmological_simulation = 0.0 yt : [INFO ] 2014-06-05 21:29:06,188 Allocating for 1.342e+08 particles
--------------------------------------------------------------------------- YTDomainOverflow Traceback (most recent call last) /nfs/blank/h4231/bgriffen/data/lib/yt-x86_64/lib/python2.7/site-packages/IPython/utils/py3compat.pyc in execfile(fname, *where) 202 else: 203 filename = fname --> 204 __builtin__.execfile(filename, *where)
/nfs/blank/h4231/bgriffen/work/projects/caterpillar/c2ray/cic/ytcic.py in <module>() 52 53 ad = ds.all_data() ---> 54 print ad['deposit', 'all_cic'] 55 slc = yt.SlicePlot(ds, 2, ('deposit', 'all_cic')) 56 slc.set_figure_size(4)
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc in __getitem__(self, key) 205 Returns a single field. Will add if necessary. 206 """ --> 207 f = self._determine_fields([key])[0] 208 if f not in self.field_data and key not in self.field_data: 209 if f in self._container_fields:
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc in _determine_fields(self, fields) 453 raise YTFieldNotParseable(field) 454 ftype, fname = field --> 455 finfo = self.pf._get_field_info(ftype, fname) 456 else: 457 fname = field
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/static_output.pyc in _get_field_info(self, ftype, fname) 445 _last_finfo = None 446 def _get_field_info(self, ftype, fname = None): --> 447 self.index 448 if fname is None: 449 ftype, fname = "unknown", ftype
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/static_output.pyc in index(self) 277 raise RuntimeError("You should not instantiate Dataset.") 278 self._instantiated_index = self._index_class( --> 279 self, dataset_type=self.dataset_type) 280 # Now we do things that we need an instantiated index for 281 # ...first off, we create our field_info now.
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/frontends/stream/data_structures.pyc in __init__(self, pf, dataset_type) 942 def __init__(self, pf, dataset_type = None): 943 self.stream_handler = pf.stream_handler --> 944 super(StreamParticleIndex, self).__init__(pf, dataset_type) 945 946 def _setup_data_io(self):
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/geometry/particle_geometry_handler.pyc in __init__(self, pf, dataset_type) 48 self.directory = os.path.dirname(self.index_filename) 49 self.float_type = np.float64 ---> 50 super(ParticleIndex, self).__init__(pf, dataset_type) 51 52 def _setup_geometry(self):
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/geometry/geometry_handler.pyc in __init__(self, pf, dataset_type) 54 55 mylog.debug("Setting up domain geometry.") ---> 56 self._setup_geometry() 57 58 mylog.debug("Initializing data grid data IO")
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/geometry/particle_geometry_handler.pyc in _setup_geometry(self) 52 def _setup_geometry(self): 53 mylog.debug("Initializing Particle Geometry Handler.") ---> 54 self._initialize_particle_handler() 55 56
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/geometry/particle_geometry_handler.pyc in _initialize_particle_handler(self) 87 pf.domain_left_edge, pf.domain_right_edge, 88 [N, N, N], len(self.data_files)) ---> 89 self._initialize_indices() 90 self.oct_handler.finalize() 91 self.max_level = self.oct_handler.max_level
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/geometry/particle_geometry_handler.pyc in _initialize_indices(self) 109 npart = sum(data_file.total_particles.values()) 110 morton[ind:ind + npart] = \ --> 111 self.io._initialize_index(data_file, self.regions) 112 ind += npart 113 morton.sort()
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/frontends/stream/io.pyc in _initialize_index(self, data_file, regions) 144 raise YTDomainOverflow(pos.min(axis=0), pos.max(axis=0), 145 data_file.pf.domain_left_edge, --> 146 data_file.pf.domain_right_edge) 147 regions.add_data_file(pos, data_file.file_id) 148 morton.append(compute_morton(
YTDomainOverflow: Particle bounds [ 0. 0. 0.] and [ 99.99999237 99.99999237 99.99999237] exceed domain bounds [ 0. 0. 0.] code_length and [ 1. 1. 1.] code_length
On Thu, Jun 5, 2014 at 8:22 PM, Nathan Goldbaum <nathan12343@gmail.com> wrote:
Here's a worked out example that does what you're looking for using a fake 1 million particle dataset:
http://nbviewer.ipython.org/gist/ngoldbaum/546d37869aafe71cfe38
In this notebook I make use of two key yt features: `load_particles`, and `covering_grid`.
load_particles creates a "stream" dataset based on in-memory data fed in as a numpy array. This dataset acts just like an on-disk simulation dataset, but doesn't come with the baggage of needing to write a custom frontend to read a specific data format off disk.
covering_grid is a way to generate uniform resolution data from an AMR dataset. It acts like a python dictionary where the keys are field names and returns 3D numpy arrays of whatever uniform resolution you specify when you create the covering_grid.
Note that if you're using load_particles all of your data needs to live in memory. If your data is too big for that you'll need to write a frontend for your data format or use a memmap to an on-disk file somehow. I'm not an expert on that but others on the list should be able to help out.
Hope that gets you well on your way :)
-Nathan
On Thu, Jun 5, 2014 at 5:04 PM, Desika Narayanan < dnarayan@haverford.edu> wrote:
Hey Brendan,
A couple of extra tools you might find helpful in conjunction with Nathan's example of depositing the particles onto an octree are at:
http://paste.yt-project.org/show/4737/
Where I load a gadget snapshot, and then recover the coordinates and width of each cell.
In response to your last question - the particles are deposited into an octree grid (so, you'll see that the cell sizes aren't all the same size). I don't know if depositing onto a regular NxNxN mesh is possible, though would be interested to hear if so.
-d
On Thu, Jun 5, 2014 at 7:58 PM, Brendan Griffen < brendan.f.griffen@gmail.com> wrote:
> Thanks. I'll get the "bleeding edge" version first then try your > suggestions. Though I want to return the NxNxN array and be able to write > this mesh to a file. It is *only* using the cic part of yt and it should > return the mesh to be written? Just wanted to clarify? > > Thanks. > Brendan > > > On Thu, Jun 5, 2014 at 6:49 PM, Nathan Goldbaum < > nathan12343@gmail.com> wrote: > >> >> >> >> On Thu, Jun 5, 2014 at 3:36 PM, John ZuHone <jzuhone@gmail.com> >> wrote: >> >>> Hi Brendan, >>> >>> Which version of yt are you using? >>> >>> If you're using 3.0, this is actually fairly easy. If you look in >>> yt.fields.particle_fields.py, around line 85, you can see how >>> this is done for the "particle_density" and "particle_mass" fields. >>> Basically you can call a "deposit" method which takes the particle field >>> quantity you want deposited and deposits it into cells. The underlying >>> calculation is done using Cython, so it's fast. >>> >> >> And you shouldn't ever actually need to call these "deposit" >> functions, since "deposit" is exposed as a field type for all datasets that >> contain particles. >> >> Here is a notebook that does this for Enzo AMR data: >> >> http://nbviewer.ipython.org/gist/ngoldbaum/5e19e4e6cc2bf330149c >> >> This dataset contains about a million particles and generates a CIC >> deposition for the whole domain in about 6 seconds from a cold start. >> >> >>> >>> If you're using 2.x, then you can do the same thing, but it's not >>> as straightforward. You can see how this works in >>> yt.data_objects.universal_fields.py, around line 986, where the >>> "particle_density" field is defined. Basically, it calls CICDeposit_3, >>> which is in yt.utilities.lib.CICDeposit.pyx. >>> >>> Let me know if you need any more clarification. >>> >>> Best, >>> >>> John Z >>> >>> On Jun 5, 2014, at 6:07 PM, Brendan Griffen < >>> brendan.f.griffen@gmail.com> wrote: >>> >>> > Hi, >>> > >>> > I was wondering if there were any Cython routines within yt >>> which takes particle data and converts it into a cloud-in-cell based mesh >>> which can be written to a file of my choosing. >>> >> >> What sort of mesh were you looking for? yt will internally >> construct an octree if it is fed particle data. I'm not sure whether this >> octree can be saved to disk for later analysis. >> >> It's also possible to create a uniform resolution covering grid >> containing field data for a deposited quantity, which can be quite easily >> saved to disk in a number of ways. >> >> >>> I heard a while ago there was some such functionality but it could >>> be too far down the yt rabbit hole to be used as a standalone? Is this >>> true? I have my own Python code for doing it but it just isn't fast enough >>> and thought I'd ask the yt community if there were any wrapper tools >>> available to boost the speed. >>> > >>> > Thanks. >>> > Brendan >>> > _______________________________________________ >>> > yt-users mailing list >>> > yt-users@lists.spacepope.org >>> > http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >>> >>> _______________________________________________ >>> yt-users mailing list >>> yt-users@lists.spacepope.org >>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >>> >> >> >> _______________________________________________ >> yt-users mailing list >> yt-users@lists.spacepope.org >> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >> >> > > _______________________________________________ > yt-users mailing list > yt-users@lists.spacepope.org > http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org > >
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
On Jun 6, 2014 4:54 PM, "Brendan Griffen" <brendan.f.griffen@gmail.com> wrote:
OK great. It is very low resolution but it worked. Thanks for all your
help. My higher resolution run 1024^3 in 100 Mpc seems to crash on 128GB memory machine. I might have to look elsewhere.
Also, I normally use Canopy distribution but I just use an alias to loadyt which erases my PYTHONPATH and I can't access scipy and a few other
If you are looking for low resolution extraction you can tune the memory usage by changing the parameter n_ref to something higher. Supporting extremely large datasets in a single mesh is on the roadmap for the late summer or fall, after a 3.0 release goes out. For now you can also extract before you load in; this is sort of how we are supporting an INCITE project with very large particle counts. libraries any more. What is the best practice here? Should I just manually export PYTHONPATH and point to the libraries need in canopy or can they play nice together?
Thanks.
BG
On Fri, Jun 6, 2014 at 2:54 PM, Nathan Goldbaum <nathan12343@gmail.com>
On Fri, Jun 6, 2014 at 11:48 AM, Brendan Griffen <
brendan.f.griffen@gmail.com> wrote:
OK great. Thanks. I just wanted a homogeneous mesh. 512^3 with no
nesting of any kind. Though when I plot the image it looks like it is assigning particles incorrectly (low resolution on the outside). This is just a test image.
The SlicePlot is visualizing the octree so there is less resolution where there are fewer particles. If you want to visualize the covering grid you're going to need to visualize that separately.
ds = yt.load_particles(data, length_unit=3.08e24,
mass_unit=1.9891e33,bbox=bbox)
ad = ds.all_data() print ad['deposit', 'all_cic'] slc = yt.SlicePlot(ds, 2, ('deposit', 'all_cic')) slc.set_figure_size(4) cg = ds.covering_grid(level=9, left_edge=[0,0,0],dims=[512,512,512])
To actually produce the uniform resolution ndarray, you're going to need to do something like:
array = cg[('deposit', 'all_cic')]
array will then be a 3D array you can do whatever you want with. By default it has units, but to strip them off you'll just need to cast to ndarray:
array_without_units = array.v
Also, is there a way to load multiple particle types?
Do I just need to stack the particles into the array here?
data = {'particle_position_x': pos[:,0], 'particle_position_y': pos[:,1], 'particle_position_z': pos[:,2], 'particle_mass': np.array([mpart]*npart)}
Then feed it in as usual?
That's right, although if the particle masses are different for the different particle types that code snippet will need to be generalized to handle that.
I think in principle it should be possible to make load_particles handle different particle types just like an SPH dataset that contains multiple
Brendan
On Thu, Jun 5, 2014 at 9:44 PM, Nathan Goldbaum <nathan12343@gmail.com>
wrote:
That's right, you can set that via the bbox keyword parameter for
load_particles. I'd urge you to take a look at the docstrings and source code for load_particles.
On Thu, Jun 5, 2014 at 6:34 PM, Brendan Griffen <
brendan.f.griffen@gmail.com> wrote:
Thanks very much Nathan. I tried to load in my own data but I think
In this area:
data = {'particle_position_x': pos[:,0], 'particle_position_y': pos[:,1], 'particle_position_z': pos[:,2], 'particle_mass': np.array([mpart]*npart)}
ds = yt.load_particles(data, length_unit=3.08e24, mass_unit=1.9891e36) ad = ds.all_data() print ad['deposit', 'all_cic']
In [3]: run ytcic.py yt : [INFO ] 2014-06-05 21:29:06,183 Parameters: current_time
= 0.0
yt : [INFO ] 2014-06-05 21:29:06,183 Parameters: domain_dimensions = [2 2 2] yt : [INFO ] 2014-06-05 21:29:06,184 Parameters: domain_left_edge = [ 0. 0. 0.] yt : [INFO ] 2014-06-05 21:29:06,185 Parameters: domain_right_edge = [ 1. 1. 1.] yt : [INFO ] 2014-06-05 21:29:06,185 Parameters: cosmological_simulation = 0.0 yt : [INFO ] 2014-06-05 21:29:06,188 Allocating for 1.342e+08
wrote: particle types, but right now that hasn't been implemented yet. there are too many particles or I have to specifically set the domain size. particles
YTDomainOverflow Traceback (most recent call last)
/nfs/blank/h4231/bgriffen/data/lib/yt-x86_64/lib/python2.7/site-packages/IPython/utils/py3compat.pyc in execfile(fname, *where)
202 else: 203 filename = fname --> 204 __builtin__.execfile(filename, *where)
/nfs/blank/h4231/bgriffen/work/projects/caterpillar/c2ray/cic/ytcic.py in <module>()
52 53 ad = ds.all_data() ---> 54 print ad['deposit', 'all_cic'] 55 slc = yt.SlicePlot(ds, 2, ('deposit', 'all_cic')) 56 slc.set_figure_size(4)
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc in __getitem__(self, key)
205 Returns a single field. Will add if necessary. 206 """ --> 207 f = self._determine_fields([key])[0] 208 if f not in self.field_data and key not in
self.field_data:
209 if f in self._container_fields:
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc in _determine_fields(self, fields)
453 raise YTFieldNotParseable(field) 454 ftype, fname = field --> 455 finfo = self.pf._get_field_info(ftype, fname) 456 else: 457 fname = field
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/static_output.pyc in _get_field_info(self, ftype, fname)
445 _last_finfo = None 446 def _get_field_info(self, ftype, fname = None): --> 447 self.index 448 if fname is None: 449 ftype, fname = "unknown", ftype
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/static_output.pyc in index(self)
277 raise RuntimeError("You should not
instantiate Dataset.")
278 self._instantiated_index = self._index_class( --> 279 self, dataset_type=self.dataset_type) 280 # Now we do things that we need an instantiated
index for
281 # ...first off, we create our field_info now.
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/frontends/stream/data_structures.pyc in __init__(self, pf, dataset_type)
942 def __init__(self, pf, dataset_type = None): 943 self.stream_handler = pf.stream_handler --> 944 super(StreamParticleIndex, self).__init__(pf,
dataset_type)
945 946 def _setup_data_io(self):
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/geometry/particle_geometry_handler.pyc in __init__(self, pf, dataset_type)
48 self.directory = os.path.dirname(self.index_filename) 49 self.float_type = np.float64 ---> 50 super(ParticleIndex, self).__init__(pf, dataset_type) 51 52 def _setup_geometry(self):
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/geometry/geometry_handler.pyc in __init__(self, pf, dataset_type)
54 55 mylog.debug("Setting up domain geometry.") ---> 56 self._setup_geometry() 57 58 mylog.debug("Initializing data grid data IO")
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/geometry/particle_geometry_handler.pyc in _setup_geometry(self)
52 def _setup_geometry(self): 53 mylog.debug("Initializing Particle Geometry Handler.") ---> 54 self._initialize_particle_handler() 55 56
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/geometry/particle_geometry_handler.pyc in _initialize_particle_handler(self)
87 pf.domain_left_edge, pf.domain_right_edge, 88 [N, N, N], len(self.data_files)) ---> 89 self._initialize_indices() 90 self.oct_handler.finalize() 91 self.max_level = self.oct_handler.max_level
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/geometry/particle_geometry_handler.pyc in _initialize_indices(self)
109 npart = sum(data_file.total_particles.values()) 110 morton[ind:ind + npart] = \ --> 111 self.io._initialize_index(data_file,
self.regions)
112 ind += npart 113 morton.sort()
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/frontends/stream/io.pyc in _initialize_index(self, data_file, regions)
144 raise YTDomainOverflow(pos.min(axis=0),
pos.max(axis=0),
145
--> 146 data_file.pf.domain_right_edge) 147 regions.add_data_file(pos, data_file.file_id) 148 morton.append(compute_morton(
YTDomainOverflow: Particle bounds [ 0. 0. 0.] and [ 99.99999237 99.99999237 99.99999237] exceed domain bounds [ 0. 0. 0.] code_length and [ 1. 1. 1.] code_length
On Thu, Jun 5, 2014 at 8:22 PM, Nathan Goldbaum <nathan12343@gmail.com> wrote:
Here's a worked out example that does what you're looking for using
a fake 1 million particle dataset:
http://nbviewer.ipython.org/gist/ngoldbaum/546d37869aafe71cfe38
In this notebook I make use of two key yt features:
`load_particles`, and `covering_grid`.
load_particles creates a "stream" dataset based on in-memory data
fed in as a numpy array. This dataset acts just like an on-disk simulation dataset, but doesn't come with the baggage of needing to write a custom frontend to read a specific data format off disk.
covering_grid is a way to generate uniform resolution data from an
AMR dataset. It acts like a python dictionary where the keys are field names and returns 3D numpy arrays of whatever uniform resolution you specify when you create the covering_grid.
Note that if you're using load_particles all of your data needs to
Hope that gets you well on your way :)
-Nathan
On Thu, Jun 5, 2014 at 5:04 PM, Desika Narayanan <
dnarayan@haverford.edu> wrote:
> > Hey Brendan, > > A couple of extra tools you might find helpful in conjunction with Nathan's example of depositing the particles onto an octree are at: > > http://paste.yt-project.org/show/4737/ > > Where I load a gadget snapshot, and then recover the coordinates and width of each cell. > > In response to your last question - the particles are deposited into an octree grid (so, you'll see that the cell sizes aren't all the same size). I don't know if depositing onto a regular NxNxN mesh is possible,
> > -d > > > On Thu, Jun 5, 2014 at 7:58 PM, Brendan Griffen < brendan.f.griffen@gmail.com> wrote: >> >> Thanks. I'll get the "bleeding edge" version first then try your suggestions. Though I want to return the NxNxN array and be able to write
>> >> Thanks. >> Brendan >> >> >> On Thu, Jun 5, 2014 at 6:49 PM, Nathan Goldbaum < nathan12343@gmail.com> wrote: >>> >>> >>> >>> >>> On Thu, Jun 5, 2014 at 3:36 PM, John ZuHone <jzuhone@gmail.com> wrote: >>>> >>>> Hi Brendan, >>>> >>>> Which version of yt are you using? >>>> >>>> If you're using 3.0, this is actually fairly easy. If you look in yt.fields.particle_fields.py, around line 85, you can see how this is done for the "particle_density" and "particle_mass" fields. Basically you can call a "deposit" method which takes the particle field quantity you want deposited and deposits it into cells. The underlying calculation is done using Cython, so it's fast. >>> >>> >>> And you shouldn't ever actually need to call these "deposit" functions, since "deposit" is exposed as a field type for all datasets that contain particles. >>> >>> Here is a notebook that does this for Enzo AMR data: >>> >>> http://nbviewer.ipython.org/gist/ngoldbaum/5e19e4e6cc2bf330149c >>> >>> This dataset contains about a million particles and generates a CIC deposition for the whole domain in about 6 seconds from a cold start. >>> >>>> >>>> >>>> If you're using 2.x, then you can do the same thing, but it's not as straightforward. You can see how this works in yt.data_objects.universal_fields.py, around line 986, where the "particle_density" field is defined. Basically, it calls CICDeposit_3, which is in yt.utilities.lib.CICDeposit.pyx. >>>> >>>> Let me know if you need any more clarification. >>>> >>>> Best, >>>> >>>> John Z >>>> >>>> On Jun 5, 2014, at 6:07 PM, Brendan Griffen < brendan.f.griffen@gmail.com> wrote: >>>> >>>> > Hi, >>>> > >>>> > I was wondering if there were any Cython routines within yt which takes particle data and converts it into a cloud-in-cell based mesh which can be written to a file of my choosing. >>> >>> >>> What sort of mesh were you looking for? yt will internally construct an octree if it is fed particle data. I'm not sure whether this octree can be saved to disk for later analysis. >>> >>> It's also possible to create a uniform resolution covering grid containing field data for a deposited quantity, which can be quite easily saved to disk in a number of ways. >>> >>>> >>>> I heard a while ago there was some such functionality but it could be too far down the yt rabbit hole to be used as a standalone? Is
data_file.pf.domain_left_edge, live in memory. If your data is too big for that you'll need to write a frontend for your data format or use a memmap to an on-disk file somehow. I'm not an expert on that but others on the list should be able to help out. though would be interested to hear if so. this mesh to a file. It is *only* using the cic part of yt and it should return the mesh to be written? Just wanted to clarify? this true? I have my own Python code for doing it but it just isn't fast enough and thought I'd ask the yt community if there were any wrapper tools available to boost the speed.
>>>> > >>>> > Thanks. >>>> > Brendan >>>> > _______________________________________________ >>>> > yt-users mailing list >>>> > yt-users@lists.spacepope.org >>>> > http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >>>> >>>> _______________________________________________ >>>> yt-users mailing list >>>> yt-users@lists.spacepope.org >>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >>> >>> >>> >>> _______________________________________________ >>> yt-users mailing list >>> yt-users@lists.spacepope.org >>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >>> >> >> >> _______________________________________________ >> yt-users mailing list >> yt-users@lists.spacepope.org >> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >> > > > _______________________________________________ > yt-users mailing list > yt-users@lists.spacepope.org > http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
Hi, Since I get memory errors. Could I not just read in the blocks of the output individually then basically stack the mesh each time. That way not every single particle of the snapshot has to be loaded at the same time. Would that just be a case of doing level = int(math.log(ndim,2)) cg = ds.covering_grid(level=level, left_edge=[0,0,0],dims=[ndim,ndim,ndim]) arr = cg['deposit', 'all_density'] arrall += arr in a loop over each HDF5 block? Thanks. Brendan On Fri, Jun 6, 2014 at 7:26 PM, Matthew Turk <matthewturk@gmail.com> wrote:
On Jun 6, 2014 4:54 PM, "Brendan Griffen" <brendan.f.griffen@gmail.com> wrote:
OK great. It is very low resolution but it worked. Thanks for all your
help. My higher resolution run 1024^3 in 100 Mpc seems to crash on 128GB memory machine. I might have to look elsewhere.
If you are looking for low resolution extraction you can tune the memory usage by changing the parameter n_ref to something higher.
Supporting extremely large datasets in a single mesh is on the roadmap for the late summer or fall, after a 3.0 release goes out. For now you can also extract before you load in; this is sort of how we are supporting an INCITE project with very large particle counts.
Also, I normally use Canopy distribution but I just use an alias to loadyt which erases my PYTHONPATH and I can't access scipy and a few other libraries any more. What is the best practice here? Should I just manually export PYTHONPATH and point to the libraries need in canopy or can they play nice together?
Thanks.
BG
On Fri, Jun 6, 2014 at 2:54 PM, Nathan Goldbaum <nathan12343@gmail.com> wrote:
On Fri, Jun 6, 2014 at 11:48 AM, Brendan Griffen <
OK great. Thanks. I just wanted a homogeneous mesh. 512^3 with no
nesting of any kind. Though when I plot the image it looks like it is assigning particles incorrectly (low resolution on the outside). This is just a test image.
The SlicePlot is visualizing the octree so there is less resolution where there are fewer particles. If you want to visualize the covering grid you're going to need to visualize that separately.
ds = yt.load_particles(data, length_unit=3.08e24,
mass_unit=1.9891e33,bbox=bbox)
ad = ds.all_data() print ad['deposit', 'all_cic'] slc = yt.SlicePlot(ds, 2, ('deposit', 'all_cic')) slc.set_figure_size(4) cg = ds.covering_grid(level=9, left_edge=[0,0,0],dims=[512,512,512])
To actually produce the uniform resolution ndarray, you're going to need to do something like:
array = cg[('deposit', 'all_cic')]
array will then be a 3D array you can do whatever you want with. By default it has units, but to strip them off you'll just need to cast to ndarray:
array_without_units = array.v
Also, is there a way to load multiple particle types?
Do I just need to stack the particles into the array here?
data = {'particle_position_x': pos[:,0], 'particle_position_y': pos[:,1], 'particle_position_z': pos[:,2], 'particle_mass': np.array([mpart]*npart)}
Then feed it in as usual?
That's right, although if the particle masses are different for the different particle types that code snippet will need to be generalized to handle that.
I think in principle it should be possible to make load_particles handle different particle types just like an SPH dataset that contains multiple particle types, but right now that hasn't been implemented yet.
Brendan
On Thu, Jun 5, 2014 at 9:44 PM, Nathan Goldbaum <nathan12343@gmail.com>
wrote:
That's right, you can set that via the bbox keyword parameter for
load_particles. I'd urge you to take a look at the docstrings and source code for load_particles.
On Thu, Jun 5, 2014 at 6:34 PM, Brendan Griffen <
brendan.f.griffen@gmail.com> wrote:
Thanks very much Nathan. I tried to load in my own data but I think
In this area:
data = {'particle_position_x': pos[:,0], 'particle_position_y': pos[:,1], 'particle_position_z': pos[:,2], 'particle_mass': np.array([mpart]*npart)}
ds = yt.load_particles(data, length_unit=3.08e24,
mass_unit=1.9891e36)
ad = ds.all_data() print ad['deposit', 'all_cic']
In [3]: run ytcic.py yt : [INFO ] 2014-06-05 21:29:06,183 Parameters: current_time = 0.0 yt : [INFO ] 2014-06-05 21:29:06,183 Parameters: domain_dimensions = [2 2 2] yt : [INFO ] 2014-06-05 21:29:06,184 Parameters: domain_left_edge = [ 0. 0. 0.] yt : [INFO ] 2014-06-05 21:29:06,185 Parameters: domain_right_edge = [ 1. 1. 1.] yt : [INFO ] 2014-06-05 21:29:06,185 Parameters: cosmological_simulation = 0.0 yt : [INFO ] 2014-06-05 21:29:06,188 Allocating for 1.342e+08
brendan.f.griffen@gmail.com> wrote: there are too many particles or I have to specifically set the domain size. particles
YTDomainOverflow Traceback (most recent call last)
/nfs/blank/h4231/bgriffen/data/lib/yt-x86_64/lib/python2.7/site-packages/IPython/utils/py3compat.pyc in execfile(fname, *where)
202 else: 203 filename = fname --> 204 __builtin__.execfile(filename, *where)
/nfs/blank/h4231/bgriffen/work/projects/caterpillar/c2ray/cic/ytcic.py in <module>()
52 53 ad = ds.all_data() ---> 54 print ad['deposit', 'all_cic'] 55 slc = yt.SlicePlot(ds, 2, ('deposit', 'all_cic')) 56 slc.set_figure_size(4)
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc in __getitem__(self, key)
205 Returns a single field. Will add if necessary. 206 """ --> 207 f = self._determine_fields([key])[0] 208 if f not in self.field_data and key not in
self.field_data:
209 if f in self._container_fields:
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc in _determine_fields(self, fields)
453 raise YTFieldNotParseable(field) 454 ftype, fname = field --> 455 finfo = self.pf._get_field_info(ftype, fname) 456 else: 457 fname = field
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/static_output.pyc in _get_field_info(self, ftype, fname)
445 _last_finfo = None 446 def _get_field_info(self, ftype, fname = None): --> 447 self.index 448 if fname is None: 449 ftype, fname = "unknown", ftype
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/static_output.pyc in index(self)
277 raise RuntimeError("You should not
instantiate Dataset.")
278 self._instantiated_index = self._index_class( --> 279 self, dataset_type=self.dataset_type) 280 # Now we do things that we need an instantiated
index for
281 # ...first off, we create our field_info now.
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/frontends/stream/data_structures.pyc in __init__(self, pf, dataset_type)
942 def __init__(self, pf, dataset_type = None): 943 self.stream_handler = pf.stream_handler --> 944 super(StreamParticleIndex, self).__init__(pf,
dataset_type)
945 946 def _setup_data_io(self):
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/geometry/particle_geometry_handler.pyc in __init__(self, pf, dataset_type)
48 self.directory = os.path.dirname(self.index_filename) 49 self.float_type = np.float64 ---> 50 super(ParticleIndex, self).__init__(pf, dataset_type) 51 52 def _setup_geometry(self):
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/geometry/geometry_handler.pyc in __init__(self, pf, dataset_type)
54 55 mylog.debug("Setting up domain geometry.") ---> 56 self._setup_geometry() 57 58 mylog.debug("Initializing data grid data IO")
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/geometry/particle_geometry_handler.pyc in _setup_geometry(self)
52 def _setup_geometry(self): 53 mylog.debug("Initializing Particle Geometry
Handler.")
---> 54 self._initialize_particle_handler() 55 56
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/geometry/particle_geometry_handler.pyc in _initialize_particle_handler(self)
87 pf.domain_left_edge, pf.domain_right_edge, 88 [N, N, N], len(self.data_files)) ---> 89 self._initialize_indices() 90 self.oct_handler.finalize() 91 self.max_level = self.oct_handler.max_level
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/geometry/particle_geometry_handler.pyc in _initialize_indices(self)
109 npart = sum(data_file.total_particles.values()) 110 morton[ind:ind + npart] = \ --> 111 self.io._initialize_index(data_file,
self.regions)
112 ind += npart 113 morton.sort()
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/frontends/stream/io.pyc in _initialize_index(self, data_file, regions)
144 raise YTDomainOverflow(pos.min(axis=0),
pos.max(axis=0),
145
--> 146 data_file.pf.domain_right_edge) 147 regions.add_data_file(pos, data_file.file_id) 148 morton.append(compute_morton(
YTDomainOverflow: Particle bounds [ 0. 0. 0.] and [ 99.99999237 99.99999237 99.99999237] exceed domain bounds [ 0. 0. 0.] code_length and [ 1. 1. 1.] code_length
On Thu, Jun 5, 2014 at 8:22 PM, Nathan Goldbaum < nathan12343@gmail.com> wrote: > > Here's a worked out example that does what you're looking for using a fake 1 million particle dataset: > > http://nbviewer.ipython.org/gist/ngoldbaum/546d37869aafe71cfe38 > > In this notebook I make use of two key yt features: `load_particles`, and `covering_grid`. > > load_particles creates a "stream" dataset based on in-memory data fed in as a numpy array. This dataset acts just like an on-disk simulation dataset, but doesn't come with the baggage of needing to write a custom frontend to read a specific data format off disk. > > covering_grid is a way to generate uniform resolution data from an AMR dataset. It acts like a python dictionary where the keys are field names and returns 3D numpy arrays of whatever uniform resolution you specify when you create the covering_grid. > > Note that if you're using load_particles all of your data needs to
> > Hope that gets you well on your way :) > > -Nathan > > > On Thu, Jun 5, 2014 at 5:04 PM, Desika Narayanan < dnarayan@haverford.edu> wrote: >> >> Hey Brendan, >> >> A couple of extra tools you might find helpful in conjunction with Nathan's example of depositing the particles onto an octree are at: >> >> http://paste.yt-project.org/show/4737/ >> >> Where I load a gadget snapshot, and then recover the coordinates and width of each cell. >> >> In response to your last question - the particles are deposited into an octree grid (so, you'll see that the cell sizes aren't all the same size). I don't know if depositing onto a regular NxNxN mesh is possible,
>> >> -d >> >> >> On Thu, Jun 5, 2014 at 7:58 PM, Brendan Griffen < brendan.f.griffen@gmail.com> wrote: >>> >>> Thanks. I'll get the "bleeding edge" version first then try your suggestions. Though I want to return the NxNxN array and be able to write
>>> >>> Thanks. >>> Brendan >>> >>> >>> On Thu, Jun 5, 2014 at 6:49 PM, Nathan Goldbaum < nathan12343@gmail.com> wrote: >>>> >>>> >>>> >>>> >>>> On Thu, Jun 5, 2014 at 3:36 PM, John ZuHone <jzuhone@gmail.com> wrote: >>>>> >>>>> Hi Brendan, >>>>> >>>>> Which version of yt are you using? >>>>> >>>>> If you're using 3.0, this is actually fairly easy. If you look in yt.fields.particle_fields.py, around line 85, you can see how this is done for the "particle_density" and "particle_mass" fields. Basically you can call a "deposit" method which takes the particle field quantity you want deposited and deposits it into cells. The underlying calculation is done using Cython, so it's fast. >>>> >>>> >>>> And you shouldn't ever actually need to call these "deposit" functions, since "deposit" is exposed as a field type for all datasets that contain particles. >>>> >>>> Here is a notebook that does this for Enzo AMR data: >>>> >>>> http://nbviewer.ipython.org/gist/ngoldbaum/5e19e4e6cc2bf330149c >>>> >>>> This dataset contains about a million particles and generates a CIC deposition for the whole domain in about 6 seconds from a cold start. >>>> >>>>> >>>>> >>>>> If you're using 2.x, then you can do the same thing, but it's not as straightforward. You can see how this works in yt.data_objects.universal_fields.py, around line 986, where the "particle_density" field is defined. Basically, it calls CICDeposit_3, which is in yt.utilities.lib.CICDeposit.pyx. >>>>> >>>>> Let me know if you need any more clarification. >>>>> >>>>> Best, >>>>> >>>>> John Z >>>>> >>>>> On Jun 5, 2014, at 6:07 PM, Brendan Griffen < brendan.f.griffen@gmail.com> wrote: >>>>> >>>>> > Hi, >>>>> > >>>>> > I was wondering if there were any Cython routines within yt which takes particle data and converts it into a cloud-in-cell based mesh which can be written to a file of my choosing. >>>> >>>> >>>> What sort of mesh were you looking for? yt will internally construct an octree if it is fed particle data. I'm not sure whether this octree can be saved to disk for later analysis. >>>> >>>> It's also possible to create a uniform resolution covering grid containing field data for a deposited quantity, which can be quite easily saved to disk in a number of ways. >>>> >>>>> >>>>> I heard a while ago there was some such functionality but it could be too far down the yt rabbit hole to be used as a standalone? Is
data_file.pf.domain_left_edge, live in memory. If your data is too big for that you'll need to write a frontend for your data format or use a memmap to an on-disk file somehow. I'm not an expert on that but others on the list should be able to help out. though would be interested to hear if so. this mesh to a file. It is *only* using the cic part of yt and it should return the mesh to be written? Just wanted to clarify? this true? I have my own Python code for doing it but it just isn't fast enough and thought I'd ask the yt community if there were any wrapper tools available to boost the speed.
>>>>> > >>>>> > Thanks. >>>>> > Brendan >>>>> > _______________________________________________ >>>>> > yt-users mailing list >>>>> > yt-users@lists.spacepope.org >>>>> > http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >>>>> >>>>> _______________________________________________ >>>>> yt-users mailing list >>>>> yt-users@lists.spacepope.org >>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >>>> >>>> >>>> >>>> _______________________________________________ >>>> yt-users mailing list >>>> yt-users@lists.spacepope.org >>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >>>> >>> >>> >>> _______________________________________________ >>> yt-users mailing list >>> yt-users@lists.spacepope.org >>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >>> >> >> >> _______________________________________________ >> yt-users mailing list >> yt-users@lists.spacepope.org >> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >> > > > _______________________________________________ > yt-users mailing list > yt-users@lists.spacepope.org > http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
Also, how do I construct just a zero filled yt array with dimensions (ndim,ndim,ndim)? Thanks Brendan On Sun, Jun 8, 2014 at 3:26 PM, Brendan Griffen <brendan.f.griffen@gmail.com
wrote:
Hi,
Since I get memory errors. Could I not just read in the blocks of the output individually then basically stack the mesh each time. That way not every single particle of the snapshot has to be loaded at the same time. Would that just be a case of doing
level = int(math.log(ndim,2)) cg = ds.covering_grid(level=level, left_edge=[0,0,0],dims=[ndim,ndim,ndim]) arr = cg['deposit', 'all_density'] arrall += arr
in a loop over each HDF5 block?
Thanks. Brendan
On Fri, Jun 6, 2014 at 7:26 PM, Matthew Turk <matthewturk@gmail.com> wrote:
On Jun 6, 2014 4:54 PM, "Brendan Griffen" <brendan.f.griffen@gmail.com> wrote:
OK great. It is very low resolution but it worked. Thanks for all your
help. My higher resolution run 1024^3 in 100 Mpc seems to crash on 128GB memory machine. I might have to look elsewhere.
If you are looking for low resolution extraction you can tune the memory usage by changing the parameter n_ref to something higher.
Supporting extremely large datasets in a single mesh is on the roadmap for the late summer or fall, after a 3.0 release goes out. For now you can also extract before you load in; this is sort of how we are supporting an INCITE project with very large particle counts.
Also, I normally use Canopy distribution but I just use an alias to loadyt which erases my PYTHONPATH and I can't access scipy and a few other libraries any more. What is the best practice here? Should I just manually export PYTHONPATH and point to the libraries need in canopy or can they play nice together?
Thanks.
BG
On Fri, Jun 6, 2014 at 2:54 PM, Nathan Goldbaum <nathan12343@gmail.com> wrote:
On Fri, Jun 6, 2014 at 11:48 AM, Brendan Griffen <
OK great. Thanks. I just wanted a homogeneous mesh. 512^3 with no
nesting of any kind. Though when I plot the image it looks like it is assigning particles incorrectly (low resolution on the outside). This is just a test image.
The SlicePlot is visualizing the octree so there is less resolution where there are fewer particles. If you want to visualize the covering grid you're going to need to visualize that separately.
ds = yt.load_particles(data, length_unit=3.08e24,
mass_unit=1.9891e33,bbox=bbox)
ad = ds.all_data() print ad['deposit', 'all_cic'] slc = yt.SlicePlot(ds, 2, ('deposit', 'all_cic')) slc.set_figure_size(4) cg = ds.covering_grid(level=9, left_edge=[0,0,0],dims=[512,512,512])
To actually produce the uniform resolution ndarray, you're going to need to do something like:
array = cg[('deposit', 'all_cic')]
array will then be a 3D array you can do whatever you want with. By default it has units, but to strip them off you'll just need to cast to ndarray:
array_without_units = array.v
Also, is there a way to load multiple particle types?
Do I just need to stack the particles into the array here?
data = {'particle_position_x': pos[:,0], 'particle_position_y': pos[:,1], 'particle_position_z': pos[:,2], 'particle_mass': np.array([mpart]*npart)}
Then feed it in as usual?
That's right, although if the particle masses are different for the different particle types that code snippet will need to be generalized to handle that.
I think in principle it should be possible to make load_particles handle different particle types just like an SPH dataset that contains multiple particle types, but right now that hasn't been implemented yet.
Brendan
On Thu, Jun 5, 2014 at 9:44 PM, Nathan Goldbaum <
nathan12343@gmail.com> wrote:
That's right, you can set that via the bbox keyword parameter for
load_particles. I'd urge you to take a look at the docstrings and source code for load_particles.
On Thu, Jun 5, 2014 at 6:34 PM, Brendan Griffen <
brendan.f.griffen@gmail.com> wrote:
> > Thanks very much Nathan. I tried to load in my own data but I think
> > In this area: > > data = {'particle_position_x': pos[:,0], > 'particle_position_y': pos[:,1], > 'particle_position_z': pos[:,2], > 'particle_mass': np.array([mpart]*npart)} > > ds = yt.load_particles(data, length_unit=3.08e24, mass_unit=1.9891e36) > ad = ds.all_data() > print ad['deposit', 'all_cic'] > > In [3]: run ytcic.py > yt : [INFO ] 2014-06-05 21:29:06,183 Parameters: current_time = 0.0 > yt : [INFO ] 2014-06-05 21:29:06,183 Parameters: domain_dimensions = [2 2 2] > yt : [INFO ] 2014-06-05 21:29:06,184 Parameters: domain_left_edge = [ 0. 0. 0.] > yt : [INFO ] 2014-06-05 21:29:06,185 Parameters: domain_right_edge = [ 1. 1. 1.] > yt : [INFO ] 2014-06-05 21:29:06,185 Parameters: cosmological_simulation = 0.0 > yt : [INFO ] 2014-06-05 21:29:06,188 Allocating for 1.342e+08
brendan.f.griffen@gmail.com> wrote: there are too many particles or I have to specifically set the domain size. particles
>
> YTDomainOverflow Traceback (most recent call last) > /nfs/blank/h4231/bgriffen/data/lib/yt-x86_64/lib/python2.7/site-packages/IPython/utils/py3compat.pyc in execfile(fname, *where) > 202 else: > 203 filename = fname > --> 204 __builtin__.execfile(filename, *where) > > /nfs/blank/h4231/bgriffen/work/projects/caterpillar/c2ray/cic/ytcic.py in <module>() > 52 > 53 ad = ds.all_data() > ---> 54 print ad['deposit', 'all_cic'] > 55 slc = yt.SlicePlot(ds, 2, ('deposit', 'all_cic')) > 56 slc.set_figure_size(4) > > /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc in __getitem__(self, key) > 205 Returns a single field. Will add if necessary. > 206 """ > --> 207 f = self._determine_fields([key])[0] > 208 if f not in self.field_data and key not in self.field_data: > 209 if f in self._container_fields: > > /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc in _determine_fields(self, fields) > 453 raise YTFieldNotParseable(field) > 454 ftype, fname = field > --> 455 finfo = self.pf._get_field_info(ftype, fname) > 456 else: > 457 fname = field > > /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/static_output.pyc in _get_field_info(self, ftype, fname) > 445 _last_finfo = None > 446 def _get_field_info(self, ftype, fname = None): > --> 447 self.index > 448 if fname is None: > 449 ftype, fname = "unknown", ftype > > /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/static_output.pyc in index(self) > 277 raise RuntimeError("You should not instantiate Dataset.") > 278 self._instantiated_index = self._index_class( > --> 279 self, dataset_type=self.dataset_type) > 280 # Now we do things that we need an instantiated index for > 281 # ...first off, we create our field_info now. > > /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/frontends/stream/data_structures.pyc in __init__(self, pf, dataset_type) > 942 def __init__(self, pf, dataset_type = None): > 943 self.stream_handler = pf.stream_handler > --> 944 super(StreamParticleIndex, self).__init__(pf, dataset_type) > 945 > 946 def _setup_data_io(self): > > /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/geometry/particle_geometry_handler.pyc in __init__(self, pf, dataset_type) > 48 self.directory = os.path.dirname(self.index_filename) > 49 self.float_type = np.float64 > ---> 50 super(ParticleIndex, self).__init__(pf, dataset_type) > 51 > 52 def _setup_geometry(self): > > /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/geometry/geometry_handler.pyc in __init__(self, pf, dataset_type) > 54 > 55 mylog.debug("Setting up domain geometry.") > ---> 56 self._setup_geometry() > 57 > 58 mylog.debug("Initializing data grid data IO") > > /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/geometry/particle_geometry_handler.pyc in _setup_geometry(self) > 52 def _setup_geometry(self): > 53 mylog.debug("Initializing Particle Geometry Handler.") > ---> 54 self._initialize_particle_handler() > 55 > 56 > > /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/geometry/particle_geometry_handler.pyc in _initialize_particle_handler(self) > 87 pf.domain_left_edge, pf.domain_right_edge, > 88 [N, N, N], len(self.data_files)) > ---> 89 self._initialize_indices() > 90 self.oct_handler.finalize() > 91 self.max_level = self.oct_handler.max_level > > /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/geometry/particle_geometry_handler.pyc in _initialize_indices(self) > 109 npart = sum(data_file.total_particles.values()) > 110 morton[ind:ind + npart] = \ > --> 111 self.io._initialize_index(data_file, self.regions) > 112 ind += npart > 113 morton.sort() > > /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/frontends/stream/io.pyc in _initialize_index(self, data_file, regions) > 144 raise YTDomainOverflow(pos.min(axis=0), pos.max(axis=0), > 145 data_file.pf.domain_left_edge, > --> 146 data_file.pf.domain_right_edge) > 147 regions.add_data_file(pos, data_file.file_id) > 148 morton.append(compute_morton( > > YTDomainOverflow: Particle bounds [ 0. 0. 0.] and [ 99.99999237 99.99999237 99.99999237] exceed domain bounds [ 0. 0. 0.] code_length and [ 1. 1. 1.] code_length > > > > On Thu, Jun 5, 2014 at 8:22 PM, Nathan Goldbaum < nathan12343@gmail.com> wrote: >> >> Here's a worked out example that does what you're looking for using a fake 1 million particle dataset: >> >> http://nbviewer.ipython.org/gist/ngoldbaum/546d37869aafe71cfe38 >> >> In this notebook I make use of two key yt features: `load_particles`, and `covering_grid`. >> >> load_particles creates a "stream" dataset based on in-memory data fed in as a numpy array. This dataset acts just like an on-disk simulation dataset, but doesn't come with the baggage of needing to write a custom frontend to read a specific data format off disk. >> >> covering_grid is a way to generate uniform resolution data from an AMR dataset. It acts like a python dictionary where the keys are field names and returns 3D numpy arrays of whatever uniform resolution you specify when you create the covering_grid. >> >> Note that if you're using load_particles all of your data needs to live in memory. If your data is too big for that you'll need to write a frontend for your data format or use a memmap to an on-disk file somehow. I'm not an expert on that but others on the list should be able to help out. >> >> Hope that gets you well on your way :) >> >> -Nathan >> >> >> On Thu, Jun 5, 2014 at 5:04 PM, Desika Narayanan < dnarayan@haverford.edu> wrote: >>> >>> Hey Brendan, >>> >>> A couple of extra tools you might find helpful in conjunction with Nathan's example of depositing the particles onto an octree are at: >>> >>> http://paste.yt-project.org/show/4737/ >>> >>> Where I load a gadget snapshot, and then recover the coordinates and width of each cell. >>> >>> In response to your last question - the particles are deposited into an octree grid (so, you'll see that the cell sizes aren't all the same size). I don't know if depositing onto a regular NxNxN mesh is possible, though would be interested to hear if so. >>> >>> -d >>> >>> >>> On Thu, Jun 5, 2014 at 7:58 PM, Brendan Griffen < brendan.f.griffen@gmail.com> wrote: >>>> >>>> Thanks. I'll get the "bleeding edge" version first then try your suggestions. Though I want to return the NxNxN array and be able to write this mesh to a file. It is *only* using the cic part of yt and it should return the mesh to be written? Just wanted to clarify? >>>> >>>> Thanks. >>>> Brendan >>>> >>>> >>>> On Thu, Jun 5, 2014 at 6:49 PM, Nathan Goldbaum < nathan12343@gmail.com> wrote: >>>>> >>>>> >>>>> >>>>> >>>>> On Thu, Jun 5, 2014 at 3:36 PM, John ZuHone <jzuhone@gmail.com> wrote: >>>>>> >>>>>> Hi Brendan, >>>>>> >>>>>> Which version of yt are you using? >>>>>> >>>>>> If you're using 3.0, this is actually fairly easy. If you look in yt.fields.particle_fields.py, around line 85, you can see how this is done for the "particle_density" and "particle_mass" fields. Basically you can call a "deposit" method which takes the particle field quantity you want deposited and deposits it into cells. The underlying calculation is done using Cython, so it's fast. >>>>> >>>>> >>>>> And you shouldn't ever actually need to call these "deposit" functions, since "deposit" is exposed as a field type for all datasets that contain particles. >>>>> >>>>> Here is a notebook that does this for Enzo AMR data: >>>>> >>>>> http://nbviewer.ipython.org/gist/ngoldbaum/5e19e4e6cc2bf330149c >>>>> >>>>> This dataset contains about a million particles and generates a CIC deposition for the whole domain in about 6 seconds from a cold start. >>>>> >>>>>> >>>>>> >>>>>> If you're using 2.x, then you can do the same thing, but it's not as straightforward. You can see how this works in yt.data_objects.universal_fields.py, around line 986, where the "particle_density" field is defined. Basically, it calls CICDeposit_3, which is in yt.utilities.lib.CICDeposit.pyx. >>>>>> >>>>>> Let me know if you need any more clarification. >>>>>> >>>>>> Best, >>>>>> >>>>>> John Z >>>>>> >>>>>> On Jun 5, 2014, at 6:07 PM, Brendan Griffen < brendan.f.griffen@gmail.com> wrote: >>>>>> >>>>>> > Hi, >>>>>> > >>>>>> > I was wondering if there were any Cython routines within yt which takes particle data and converts it into a cloud-in-cell based mesh which can be written to a file of my choosing. >>>>> >>>>> >>>>> What sort of mesh were you looking for? yt will internally construct an octree if it is fed particle data. I'm not sure whether this octree can be saved to disk for later analysis. >>>>> >>>>> It's also possible to create a uniform resolution covering grid containing field data for a deposited quantity, which can be quite easily saved to disk in a number of ways. >>>>> >>>>>> >>>>>> I heard a while ago there was some such functionality but it could be too far down the yt rabbit hole to be used as a standalone? Is this true? I have my own Python code for doing it but it just isn't fast enough and thought I'd ask the yt community if there were any wrapper tools available to boost the speed. >>>>>> > >>>>>> > Thanks. >>>>>> > Brendan >>>>>> > _______________________________________________ >>>>>> > yt-users mailing list >>>>>> > yt-users@lists.spacepope.org >>>>>> > http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >>>>>> >>>>>> _______________________________________________ >>>>>> yt-users mailing list >>>>>> yt-users@lists.spacepope.org >>>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> yt-users mailing list >>>>> yt-users@lists.spacepope.org >>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >>>>> >>>> >>>> >>>> _______________________________________________ >>>> yt-users mailing list >>>> yt-users@lists.spacepope.org >>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >>>> >>> >>> >>> _______________________________________________ >>> yt-users mailing list >>> yt-users@lists.spacepope.org >>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >>> >> >> >> _______________________________________________ >> yt-users mailing list >> yt-users@lists.spacepope.org >> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >> > > > _______________________________________________ > yt-users mailing list > yt-users@lists.spacepope.org > http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
On Sun, Jun 8, 2014 at 12:27 PM, Brendan Griffen < brendan.f.griffen@gmail.com> wrote:
Also, how do I construct just a zero filled yt array with dimensions (ndim,ndim,ndim)? Thanks
from yt import YTArray from numpy import np arr = YTArray(np.zeros([ndim, ndim, ndim]), input_units=units_string) or alternatively: from yt.units import kiloparsec arr = kiloparsec*np.zeros([ndim, ndim, ndim]) it doesn't have to be kiloparsec - you can compose the units you want out of any of the unit symbols that live in yt.units. See this page for a ton more detail about yt's new unit system: http://yt-project.org/docs/dev-3.0/analyzing/units/index.html
Brendan
On Sun, Jun 8, 2014 at 3:26 PM, Brendan Griffen < brendan.f.griffen@gmail.com> wrote:
Hi,
Since I get memory errors. Could I not just read in the blocks of the output individually then basically stack the mesh each time. That way not every single particle of the snapshot has to be loaded at the same time. Would that just be a case of doing
level = int(math.log(ndim,2)) cg = ds.covering_grid(level=level, left_edge=[0,0,0],dims=[ndim,ndim,ndim]) arr = cg['deposit', 'all_density'] arrall += arr
in a loop over each HDF5 block?
It's likely that the memory use is dominated by the octree rather than the covering grid. This is with 1024^3 particles, correct? You can probably significantly reduce the memory used by the octree by increasing n_ref in the call to load_particles. See this page for more detail about load_particles: http://yt-project.org/docs/dev-3.0/examining/loading_data.html#generic-parti... Larger n_ref means fewer octree cells (lower resolution), but it also means lower poisson noise and lower memory use. Alternatively, as Matt suggested, you could break your 1024^3 ensemble of particles up into chunks, loop over the chunk, creating a particle octree and then a covering grid for each subset of the particles. Your final covering grid is just the sub of the covering grids for each subset of particles.
Thanks. Brendan
On Fri, Jun 6, 2014 at 7:26 PM, Matthew Turk <matthewturk@gmail.com> wrote:
On Jun 6, 2014 4:54 PM, "Brendan Griffen" <brendan.f.griffen@gmail.com> wrote:
OK great. It is very low resolution but it worked. Thanks for all your
help. My higher resolution run 1024^3 in 100 Mpc seems to crash on 128GB memory machine. I might have to look elsewhere.
If you are looking for low resolution extraction you can tune the memory usage by changing the parameter n_ref to something higher.
Supporting extremely large datasets in a single mesh is on the roadmap for the late summer or fall, after a 3.0 release goes out. For now you can also extract before you load in; this is sort of how we are supporting an INCITE project with very large particle counts.
Also, I normally use Canopy distribution but I just use an alias to loadyt which erases my PYTHONPATH and I can't access scipy and a few other libraries any more. What is the best practice here? Should I just manually export PYTHONPATH and point to the libraries need in canopy or can they play nice together?
Thanks.
BG
On Fri, Jun 6, 2014 at 2:54 PM, Nathan Goldbaum <nathan12343@gmail.com> wrote:
On Fri, Jun 6, 2014 at 11:48 AM, Brendan Griffen <
OK great. Thanks. I just wanted a homogeneous mesh. 512^3 with no
nesting of any kind. Though when I plot the image it looks like it is assigning particles incorrectly (low resolution on the outside). This is just a test image.
The SlicePlot is visualizing the octree so there is less resolution where there are fewer particles. If you want to visualize the covering grid you're going to need to visualize that separately.
ds = yt.load_particles(data, length_unit=3.08e24,
mass_unit=1.9891e33,bbox=bbox)
ad = ds.all_data() print ad['deposit', 'all_cic'] slc = yt.SlicePlot(ds, 2, ('deposit', 'all_cic')) slc.set_figure_size(4) cg = ds.covering_grid(level=9, left_edge=[0,0,0],dims=[512,512,512])
To actually produce the uniform resolution ndarray, you're going to need to do something like:
array = cg[('deposit', 'all_cic')]
array will then be a 3D array you can do whatever you want with. By default it has units, but to strip them off you'll just need to cast to ndarray:
array_without_units = array.v
Also, is there a way to load multiple particle types?
Do I just need to stack the particles into the array here?
data = {'particle_position_x': pos[:,0], 'particle_position_y': pos[:,1], 'particle_position_z': pos[:,2], 'particle_mass': np.array([mpart]*npart)}
Then feed it in as usual?
That's right, although if the particle masses are different for the different particle types that code snippet will need to be generalized to handle that.
I think in principle it should be possible to make load_particles handle different particle types just like an SPH dataset that contains multiple particle types, but right now that hasn't been implemented yet.
Brendan
On Thu, Jun 5, 2014 at 9:44 PM, Nathan Goldbaum <
nathan12343@gmail.com> wrote:
> > That's right, you can set that via the bbox keyword parameter for load_particles. I'd urge you to take a look at the docstrings and source code for load_particles. > > > On Thu, Jun 5, 2014 at 6:34 PM, Brendan Griffen < brendan.f.griffen@gmail.com> wrote: >> >> Thanks very much Nathan. I tried to load in my own data but I
>> >> In this area: >> >> data = {'particle_position_x': pos[:,0], >> 'particle_position_y': pos[:,1], >> 'particle_position_z': pos[:,2], >> 'particle_mass': np.array([mpart]*npart)} >> >> ds = yt.load_particles(data, length_unit=3.08e24, mass_unit=1.9891e36) >> ad = ds.all_data() >> print ad['deposit', 'all_cic'] >> >> In [3]: run ytcic.py >> yt : [INFO ] 2014-06-05 21:29:06,183 Parameters: current_time = 0.0 >> yt : [INFO ] 2014-06-05 21:29:06,183 Parameters: domain_dimensions = [2 2 2] >> yt : [INFO ] 2014-06-05 21:29:06,184 Parameters: domain_left_edge = [ 0. 0. 0.] >> yt : [INFO ] 2014-06-05 21:29:06,185 Parameters: domain_right_edge = [ 1. 1. 1.] >> yt : [INFO ] 2014-06-05 21:29:06,185 Parameters: cosmological_simulation = 0.0 >> yt : [INFO ] 2014-06-05 21:29:06,188 Allocating for 1.342e+08
brendan.f.griffen@gmail.com> wrote: think there are too many particles or I have to specifically set the domain size. particles
>>
>> YTDomainOverflow Traceback (most recent call last) >> /nfs/blank/h4231/bgriffen/data/lib/yt-x86_64/lib/python2.7/site-packages/IPython/utils/py3compat.pyc in execfile(fname, *where) >> 202 else: >> 203 filename = fname >> --> 204 __builtin__.execfile(filename, *where) >> >> /nfs/blank/h4231/bgriffen/work/projects/caterpillar/c2ray/cic/ytcic.py in <module>() >> 52 >> 53 ad = ds.all_data() >> ---> 54 print ad['deposit', 'all_cic'] >> 55 slc = yt.SlicePlot(ds, 2, ('deposit', 'all_cic')) >> 56 slc.set_figure_size(4) >> >> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc in __getitem__(self, key) >> 205 Returns a single field. Will add if necessary. >> 206 """ >> --> 207 f = self._determine_fields([key])[0] >> 208 if f not in self.field_data and key not in self.field_data: >> 209 if f in self._container_fields: >> >> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc in _determine_fields(self, fields) >> 453 raise YTFieldNotParseable(field) >> 454 ftype, fname = field >> --> 455 finfo = self.pf._get_field_info(ftype, fname) >> 456 else: >> 457 fname = field >> >> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/static_output.pyc in _get_field_info(self, ftype, fname) >> 445 _last_finfo = None >> 446 def _get_field_info(self, ftype, fname = None): >> --> 447 self.index >> 448 if fname is None: >> 449 ftype, fname = "unknown", ftype >> >> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/static_output.pyc in index(self) >> 277 raise RuntimeError("You should not instantiate Dataset.") >> 278 self._instantiated_index = self._index_class( >> --> 279 self, dataset_type=self.dataset_type) >> 280 # Now we do things that we need an instantiated index for >> 281 # ...first off, we create our field_info now. >> >> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/frontends/stream/data_structures.pyc in __init__(self, pf, dataset_type) >> 942 def __init__(self, pf, dataset_type = None): >> 943 self.stream_handler = pf.stream_handler >> --> 944 super(StreamParticleIndex, self).__init__(pf, dataset_type) >> 945 >> 946 def _setup_data_io(self): >> >> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/geometry/particle_geometry_handler.pyc in __init__(self, pf, dataset_type) >> 48 self.directory = os.path.dirname(self.index_filename) >> 49 self.float_type = np.float64 >> ---> 50 super(ParticleIndex, self).__init__(pf, dataset_type) >> 51 >> 52 def _setup_geometry(self): >> >> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/geometry/geometry_handler.pyc in __init__(self, pf, dataset_type) >> 54 >> 55 mylog.debug("Setting up domain geometry.") >> ---> 56 self._setup_geometry() >> 57 >> 58 mylog.debug("Initializing data grid data IO") >> >> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/geometry/particle_geometry_handler.pyc in _setup_geometry(self) >> 52 def _setup_geometry(self): >> 53 mylog.debug("Initializing Particle Geometry Handler.") >> ---> 54 self._initialize_particle_handler() >> 55 >> 56 >> >> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/geometry/particle_geometry_handler.pyc in _initialize_particle_handler(self) >> 87 pf.domain_left_edge, pf.domain_right_edge, >> 88 [N, N, N], len(self.data_files)) >> ---> 89 self._initialize_indices() >> 90 self.oct_handler.finalize() >> 91 self.max_level = self.oct_handler.max_level >> >> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/geometry/particle_geometry_handler.pyc in _initialize_indices(self) >> 109 npart = sum(data_file.total_particles.values()) >> 110 morton[ind:ind + npart] = \ >> --> 111 self.io._initialize_index(data_file, self.regions) >> 112 ind += npart >> 113 morton.sort() >> >> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/frontends/stream/io.pyc in _initialize_index(self, data_file, regions) >> 144 raise YTDomainOverflow(pos.min(axis=0), pos.max(axis=0), >> 145 data_file.pf.domain_left_edge, >> --> 146 data_file.pf.domain_right_edge) >> 147 regions.add_data_file(pos, data_file.file_id) >> 148 morton.append(compute_morton( >> >> YTDomainOverflow: Particle bounds [ 0. 0. 0.] and [ 99.99999237 99.99999237 99.99999237] exceed domain bounds [ 0. 0. 0.] code_length and [ 1. 1. 1.] code_length >> >> >> >> On Thu, Jun 5, 2014 at 8:22 PM, Nathan Goldbaum < nathan12343@gmail.com> wrote: >>> >>> Here's a worked out example that does what you're looking for using a fake 1 million particle dataset: >>> >>> http://nbviewer.ipython.org/gist/ngoldbaum/546d37869aafe71cfe38 >>> >>> In this notebook I make use of two key yt features: `load_particles`, and `covering_grid`. >>> >>> load_particles creates a "stream" dataset based on in-memory data fed in as a numpy array. This dataset acts just like an on-disk simulation dataset, but doesn't come with the baggage of needing to write a custom frontend to read a specific data format off disk. >>> >>> covering_grid is a way to generate uniform resolution data from an AMR dataset. It acts like a python dictionary where the keys are field names and returns 3D numpy arrays of whatever uniform resolution you specify when you create the covering_grid. >>> >>> Note that if you're using load_particles all of your data needs to live in memory. If your data is too big for that you'll need to write a frontend for your data format or use a memmap to an on-disk file somehow. I'm not an expert on that but others on the list should be able to help out. >>> >>> Hope that gets you well on your way :) >>> >>> -Nathan >>> >>> >>> On Thu, Jun 5, 2014 at 5:04 PM, Desika Narayanan < dnarayan@haverford.edu> wrote: >>>> >>>> Hey Brendan, >>>> >>>> A couple of extra tools you might find helpful in conjunction with Nathan's example of depositing the particles onto an octree are at: >>>> >>>> http://paste.yt-project.org/show/4737/ >>>> >>>> Where I load a gadget snapshot, and then recover the coordinates and width of each cell. >>>> >>>> In response to your last question - the particles are deposited into an octree grid (so, you'll see that the cell sizes aren't all the same size). I don't know if depositing onto a regular NxNxN mesh is possible, though would be interested to hear if so. >>>> >>>> -d >>>> >>>> >>>> On Thu, Jun 5, 2014 at 7:58 PM, Brendan Griffen < brendan.f.griffen@gmail.com> wrote: >>>>> >>>>> Thanks. I'll get the "bleeding edge" version first then try your suggestions. Though I want to return the NxNxN array and be able to write this mesh to a file. It is *only* using the cic part of yt and it should return the mesh to be written? Just wanted to clarify? >>>>> >>>>> Thanks. >>>>> Brendan >>>>> >>>>> >>>>> On Thu, Jun 5, 2014 at 6:49 PM, Nathan Goldbaum < nathan12343@gmail.com> wrote: >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Thu, Jun 5, 2014 at 3:36 PM, John ZuHone <jzuhone@gmail.com> wrote: >>>>>>> >>>>>>> Hi Brendan, >>>>>>> >>>>>>> Which version of yt are you using? >>>>>>> >>>>>>> If you're using 3.0, this is actually fairly easy. If you look in yt.fields.particle_fields.py, around line 85, you can see how this is done for the "particle_density" and "particle_mass" fields. Basically you can call a "deposit" method which takes the particle field quantity you want deposited and deposits it into cells. The underlying calculation is done using Cython, so it's fast. >>>>>> >>>>>> >>>>>> And you shouldn't ever actually need to call these "deposit" functions, since "deposit" is exposed as a field type for all datasets that contain particles. >>>>>> >>>>>> Here is a notebook that does this for Enzo AMR data: >>>>>> >>>>>> http://nbviewer.ipython.org/gist/ngoldbaum/5e19e4e6cc2bf330149c >>>>>> >>>>>> This dataset contains about a million particles and generates a CIC deposition for the whole domain in about 6 seconds from a cold start. >>>>>> >>>>>>> >>>>>>> >>>>>>> If you're using 2.x, then you can do the same thing, but it's not as straightforward. You can see how this works in yt.data_objects.universal_fields.py, around line 986, where the "particle_density" field is defined. Basically, it calls CICDeposit_3, which is in yt.utilities.lib.CICDeposit.pyx. >>>>>>> >>>>>>> Let me know if you need any more clarification. >>>>>>> >>>>>>> Best, >>>>>>> >>>>>>> John Z >>>>>>> >>>>>>> On Jun 5, 2014, at 6:07 PM, Brendan Griffen < brendan.f.griffen@gmail.com> wrote: >>>>>>> >>>>>>> > Hi, >>>>>>> > >>>>>>> > I was wondering if there were any Cython routines within yt which takes particle data and converts it into a cloud-in-cell based mesh which can be written to a file of my choosing. >>>>>> >>>>>> >>>>>> What sort of mesh were you looking for? yt will internally construct an octree if it is fed particle data. I'm not sure whether this octree can be saved to disk for later analysis. >>>>>> >>>>>> It's also possible to create a uniform resolution covering grid containing field data for a deposited quantity, which can be quite easily saved to disk in a number of ways. >>>>>> >>>>>>> >>>>>>> I heard a while ago there was some such functionality but it could be too far down the yt rabbit hole to be used as a standalone? Is this true? I have my own Python code for doing it but it just isn't fast enough and thought I'd ask the yt community if there were any wrapper tools available to boost the speed. >>>>>>> > >>>>>>> > Thanks. >>>>>>> > Brendan >>>>>>> > _______________________________________________ >>>>>>> > yt-users mailing list >>>>>>> > yt-users@lists.spacepope.org >>>>>>> > http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >>>>>>> >>>>>>> _______________________________________________ >>>>>>> yt-users mailing list >>>>>>> yt-users@lists.spacepope.org >>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >>>>>> >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> yt-users mailing list >>>>>> yt-users@lists.spacepope.org >>>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >>>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> yt-users mailing list >>>>> yt-users@lists.spacepope.org >>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >>>>> >>>> >>>> >>>> _______________________________________________ >>>> yt-users mailing list >>>> yt-users@lists.spacepope.org >>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >>>> >>> >>> >>> _______________________________________________ >>> yt-users mailing list >>> yt-users@lists.spacepope.org >>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >>> >> >> >> _______________________________________________ >> yt-users mailing list >> yt-users@lists.spacepope.org >> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >> > > > _______________________________________________ > yt-users mailing list > yt-users@lists.spacepope.org > http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
Hi all, I feel like I owe a brief explanation of why things are tricky right now, what we're planning on doing, and how we're experimenting and developing. Presently, the particle geometry handlers build a single mesh from all particles in the dataset, along with a coarse bitmask that correlates files to regions in the domain. This requires the allocation of a single int64 array of size Nparticles, which is sorted in place and then fed into an octree construction algorithm that then spits back out the mesh. Each octree component contains 3 64-bit integers and eitehr a void pointer or a pointer to eight other octs. Increasing n_ref decreases the number of octs in this mesh; when smoothing operaitons are conducted, a second "index" mesh is created for looking up particles near mesh points. Mesh points are used for adaptive resolution smoothing and other "deposit particles on the grid somehow" operations (including SPH kernel). Anyway, because right now it requires a global mesh to be constructed, this is expensive and requires holding a 64-bit integer in memory for each particle. I think if you're loading the particles in differently there is some additional overhead as well, but I'm still a bit surprised you OOM on a 1024^3 dataset. In general, we don't *need* this global mesh; is can be constructed as required, which would speed up both the initial index phase as well as the final meshing process. I got about 50% of the way to implementing this last fall, but because of various concerns and deadlines I haven't finished it. I intend to get back to it probably in July, right after we put out a 3.0, so that we can have it in time for 3.1. In principle this will make the particle codes much more similar to ARTIO, in that the mesh will be constructed only as required and discarded when no longer required, which will make them much more memory efficient. But, getting a single mesh for extremely large data is a very high priority; right now for the 10240^3 run we've been loading up individual sub-chunks, which I want to stop doing.
From the technical perspective, these are the things that need to happen on the yt side for particle datasets to move to this "lazy" mode of loading; most of this is based on things learned from 2HOT and ARTIO, and will involve converting to a forest-of-octrees.
* Split into spatially-organized subchunks of ParticleOctreeSubset objects, such that these map 1:Nfiles, and that can be constructed on the fly. * Construct a dual-mesh of the bitmask "ParticleRegion" object that will help with identifying neighbors to a given oct cell, so that if we're inside one octree we know which neighbor octrees to grab if we need particles for smoothing things (fast boundary particle identification is later down the road) * Parallel sort of particles, or using the parallel ring function; may not be necessary after all All of this is doable, and I'd be happy to work with people if they'd like to take a shot at implementing it, but I've mostly put it on my list for post-3.0. -Matt On Sun, Jun 8, 2014 at 2:43 PM, Nathan Goldbaum <nathan12343@gmail.com> wrote:
On Sun, Jun 8, 2014 at 12:27 PM, Brendan Griffen <brendan.f.griffen@gmail.com> wrote:
Also, how do I construct just a zero filled yt array with dimensions (ndim,ndim,ndim)? Thanks
from yt import YTArray from numpy import np
arr = YTArray(np.zeros([ndim, ndim, ndim]), input_units=units_string)
or alternatively:
from yt.units import kiloparsec
arr = kiloparsec*np.zeros([ndim, ndim, ndim])
it doesn't have to be kiloparsec - you can compose the units you want out of any of the unit symbols that live in yt.units.
See this page for a ton more detail about yt's new unit system: http://yt-project.org/docs/dev-3.0/analyzing/units/index.html
Brendan
On Sun, Jun 8, 2014 at 3:26 PM, Brendan Griffen <brendan.f.griffen@gmail.com> wrote:
Hi,
Since I get memory errors. Could I not just read in the blocks of the output individually then basically stack the mesh each time. That way not every single particle of the snapshot has to be loaded at the same time. Would that just be a case of doing
level = int(math.log(ndim,2)) cg = ds.covering_grid(level=level, left_edge=[0,0,0],dims=[ndim,ndim,ndim]) arr = cg['deposit', 'all_density'] arrall += arr
in a loop over each HDF5 block?
It's likely that the memory use is dominated by the octree rather than the covering grid. This is with 1024^3 particles, correct?
You can probably significantly reduce the memory used by the octree by increasing n_ref in the call to load_particles.
See this page for more detail about load_particles: http://yt-project.org/docs/dev-3.0/examining/loading_data.html#generic-parti...
Larger n_ref means fewer octree cells (lower resolution), but it also means lower poisson noise and lower memory use.
Alternatively, as Matt suggested, you could break your 1024^3 ensemble of particles up into chunks, loop over the chunk, creating a particle octree and then a covering grid for each subset of the particles. Your final covering grid is just the sub of the covering grids for each subset of particles.
Thanks. Brendan
On Fri, Jun 6, 2014 at 7:26 PM, Matthew Turk <matthewturk@gmail.com> wrote:
On Jun 6, 2014 4:54 PM, "Brendan Griffen" <brendan.f.griffen@gmail.com> wrote:
OK great. It is very low resolution but it worked. Thanks for all your help. My higher resolution run 1024^3 in 100 Mpc seems to crash on 128GB memory machine. I might have to look elsewhere.
If you are looking for low resolution extraction you can tune the memory usage by changing the parameter n_ref to something higher.
Supporting extremely large datasets in a single mesh is on the roadmap for the late summer or fall, after a 3.0 release goes out. For now you can also extract before you load in; this is sort of how we are supporting an INCITE project with very large particle counts.
Also, I normally use Canopy distribution but I just use an alias to loadyt which erases my PYTHONPATH and I can't access scipy and a few other libraries any more. What is the best practice here? Should I just manually export PYTHONPATH and point to the libraries need in canopy or can they play nice together?
Thanks.
BG
On Fri, Jun 6, 2014 at 2:54 PM, Nathan Goldbaum <nathan12343@gmail.com> wrote:
On Fri, Jun 6, 2014 at 11:48 AM, Brendan Griffen <brendan.f.griffen@gmail.com> wrote: > > OK great. Thanks. I just wanted a homogeneous mesh. 512^3 with no > nesting of any kind. Though when I plot the image it looks like it is > assigning particles incorrectly (low resolution on the outside). This is > just a test image. >
The SlicePlot is visualizing the octree so there is less resolution where there are fewer particles. If you want to visualize the covering grid you're going to need to visualize that separately.
> > ds = yt.load_particles(data, length_unit=3.08e24, > mass_unit=1.9891e33,bbox=bbox) > > ad = ds.all_data() > print ad['deposit', 'all_cic'] > slc = yt.SlicePlot(ds, 2, ('deposit', 'all_cic')) > slc.set_figure_size(4) > cg = ds.covering_grid(level=9, left_edge=[0,0,0],dims=[512,512,512]) >
To actually produce the uniform resolution ndarray, you're going to need to do something like:
array = cg[('deposit', 'all_cic')]
array will then be a 3D array you can do whatever you want with. By default it has units, but to strip them off you'll just need to cast to ndarray:
array_without_units = array.v
> > Also, is there a way to load multiple particle types? > > Do I just need to stack the particles into the array here? > > data = {'particle_position_x': pos[:,0], > 'particle_position_y': pos[:,1], > 'particle_position_z': pos[:,2], > 'particle_mass': np.array([mpart]*npart)} > > Then feed it in as usual?
That's right, although if the particle masses are different for the different particle types that code snippet will need to be generalized to handle that.
I think in principle it should be possible to make load_particles handle different particle types just like an SPH dataset that contains multiple particle types, but right now that hasn't been implemented yet.
> > > Brendan > > > On Thu, Jun 5, 2014 at 9:44 PM, Nathan Goldbaum > <nathan12343@gmail.com> wrote: >> >> That's right, you can set that via the bbox keyword parameter for >> load_particles. I'd urge you to take a look at the docstrings and source >> code for load_particles. >> >> >> On Thu, Jun 5, 2014 at 6:34 PM, Brendan Griffen >> <brendan.f.griffen@gmail.com> wrote: >>> >>> Thanks very much Nathan. I tried to load in my own data but I >>> think there are too many particles or I have to specifically set the domain >>> size. >>> >>> In this area: >>> >>> data = {'particle_position_x': pos[:,0], >>> 'particle_position_y': pos[:,1], >>> 'particle_position_z': pos[:,2], >>> 'particle_mass': np.array([mpart]*npart)} >>> >>> ds = yt.load_particles(data, length_unit=3.08e24, >>> mass_unit=1.9891e36) >>> ad = ds.all_data() >>> print ad['deposit', 'all_cic'] >>> >>> In [3]: run ytcic.py >>> yt : [INFO ] 2014-06-05 21:29:06,183 Parameters: current_time >>> = 0.0 >>> yt : [INFO ] 2014-06-05 21:29:06,183 Parameters: >>> domain_dimensions = [2 2 2] >>> yt : [INFO ] 2014-06-05 21:29:06,184 Parameters: >>> domain_left_edge = [ 0. 0. 0.] >>> yt : [INFO ] 2014-06-05 21:29:06,185 Parameters: >>> domain_right_edge = [ 1. 1. 1.] >>> yt : [INFO ] 2014-06-05 21:29:06,185 Parameters: >>> cosmological_simulation = 0.0 >>> yt : [INFO ] 2014-06-05 21:29:06,188 Allocating for 1.342e+08 >>> particles >>> >>> --------------------------------------------------------------------------- >>> YTDomainOverflow Traceback (most recent >>> call last) >>> >>> /nfs/blank/h4231/bgriffen/data/lib/yt-x86_64/lib/python2.7/site-packages/IPython/utils/py3compat.pyc >>> in execfile(fname, *where) >>> 202 else: >>> 203 filename = fname >>> --> 204 __builtin__.execfile(filename, *where) >>> >>> >>> /nfs/blank/h4231/bgriffen/work/projects/caterpillar/c2ray/cic/ytcic.py in >>> <module>() >>> 52 >>> 53 ad = ds.all_data() >>> ---> 54 print ad['deposit', 'all_cic'] >>> 55 slc = yt.SlicePlot(ds, 2, ('deposit', 'all_cic')) >>> 56 slc.set_figure_size(4) >>> >>> >>> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc >>> in __getitem__(self, key) >>> 205 Returns a single field. Will add if necessary. >>> 206 """ >>> --> 207 f = self._determine_fields([key])[0] >>> 208 if f not in self.field_data and key not in >>> self.field_data: >>> 209 if f in self._container_fields: >>> >>> >>> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc >>> in _determine_fields(self, fields) >>> 453 raise YTFieldNotParseable(field) >>> 454 ftype, fname = field >>> --> 455 finfo = self.pf._get_field_info(ftype, >>> fname) >>> 456 else: >>> 457 fname = field >>> >>> >>> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/static_output.pyc >>> in _get_field_info(self, ftype, fname) >>> 445 _last_finfo = None >>> 446 def _get_field_info(self, ftype, fname = None): >>> --> 447 self.index >>> 448 if fname is None: >>> 449 ftype, fname = "unknown", ftype >>> >>> >>> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/static_output.pyc >>> in index(self) >>> 277 raise RuntimeError("You should not >>> instantiate Dataset.") >>> 278 self._instantiated_index = self._index_class( >>> --> 279 self, dataset_type=self.dataset_type) >>> 280 # Now we do things that we need an >>> instantiated index for >>> 281 # ...first off, we create our field_info now. >>> >>> >>> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/frontends/stream/data_structures.pyc >>> in __init__(self, pf, dataset_type) >>> 942 def __init__(self, pf, dataset_type = None): >>> 943 self.stream_handler = pf.stream_handler >>> --> 944 super(StreamParticleIndex, self).__init__(pf, >>> dataset_type) >>> 945 >>> 946 def _setup_data_io(self): >>> >>> >>> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/geometry/particle_geometry_handler.pyc >>> in __init__(self, pf, dataset_type) >>> 48 self.directory = >>> os.path.dirname(self.index_filename) >>> 49 self.float_type = np.float64 >>> ---> 50 super(ParticleIndex, self).__init__(pf, >>> dataset_type) >>> 51 >>> 52 def _setup_geometry(self): >>> >>> >>> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/geometry/geometry_handler.pyc >>> in __init__(self, pf, dataset_type) >>> 54 >>> 55 mylog.debug("Setting up domain geometry.") >>> ---> 56 self._setup_geometry() >>> 57 >>> 58 mylog.debug("Initializing data grid data IO") >>> >>> >>> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/geometry/particle_geometry_handler.pyc >>> in _setup_geometry(self) >>> 52 def _setup_geometry(self): >>> 53 mylog.debug("Initializing Particle Geometry >>> Handler.") >>> ---> 54 self._initialize_particle_handler() >>> 55 >>> 56 >>> >>> >>> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/geometry/particle_geometry_handler.pyc >>> in _initialize_particle_handler(self) >>> 87 pf.domain_left_edge, pf.domain_right_edge, >>> 88 [N, N, N], len(self.data_files)) >>> ---> 89 self._initialize_indices() >>> 90 self.oct_handler.finalize() >>> 91 self.max_level = self.oct_handler.max_level >>> >>> >>> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/geometry/particle_geometry_handler.pyc >>> in _initialize_indices(self) >>> 109 npart = >>> sum(data_file.total_particles.values()) >>> 110 morton[ind:ind + npart] = \ >>> --> 111 self.io._initialize_index(data_file, >>> self.regions) >>> 112 ind += npart >>> 113 morton.sort() >>> >>> >>> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/frontends/stream/io.pyc in >>> _initialize_index(self, data_file, regions) >>> 144 raise YTDomainOverflow(pos.min(axis=0), >>> pos.max(axis=0), >>> 145 >>> data_file.pf.domain_left_edge, >>> --> 146 >>> data_file.pf.domain_right_edge) >>> 147 regions.add_data_file(pos, data_file.file_id) >>> 148 morton.append(compute_morton( >>> >>> YTDomainOverflow: Particle bounds [ 0. 0. 0.] and [ 99.99999237 >>> 99.99999237 99.99999237] exceed domain bounds [ 0. 0. 0.] code_length and >>> [ 1. 1. 1.] code_length >>> >>> >>> >>> On Thu, Jun 5, 2014 at 8:22 PM, Nathan Goldbaum >>> <nathan12343@gmail.com> wrote: >>>> >>>> Here's a worked out example that does what you're looking for >>>> using a fake 1 million particle dataset: >>>> >>>> http://nbviewer.ipython.org/gist/ngoldbaum/546d37869aafe71cfe38 >>>> >>>> In this notebook I make use of two key yt features: >>>> `load_particles`, and `covering_grid`. >>>> >>>> load_particles creates a "stream" dataset based on in-memory data >>>> fed in as a numpy array. This dataset acts just like an on-disk simulation >>>> dataset, but doesn't come with the baggage of needing to write a custom >>>> frontend to read a specific data format off disk. >>>> >>>> covering_grid is a way to generate uniform resolution data from >>>> an AMR dataset. It acts like a python dictionary where the keys are field >>>> names and returns 3D numpy arrays of whatever uniform resolution you specify >>>> when you create the covering_grid. >>>> >>>> Note that if you're using load_particles all of your data needs >>>> to live in memory. If your data is too big for that you'll need to write a >>>> frontend for your data format or use a memmap to an on-disk file somehow. >>>> I'm not an expert on that but others on the list should be able to help out. >>>> >>>> Hope that gets you well on your way :) >>>> >>>> -Nathan >>>> >>>> >>>> On Thu, Jun 5, 2014 at 5:04 PM, Desika Narayanan >>>> <dnarayan@haverford.edu> wrote: >>>>> >>>>> Hey Brendan, >>>>> >>>>> A couple of extra tools you might find helpful in conjunction >>>>> with Nathan's example of depositing the particles onto an octree are at: >>>>> >>>>> http://paste.yt-project.org/show/4737/ >>>>> >>>>> Where I load a gadget snapshot, and then recover the coordinates >>>>> and width of each cell. >>>>> >>>>> In response to your last question - the particles are deposited >>>>> into an octree grid (so, you'll see that the cell sizes aren't all the same >>>>> size). I don't know if depositing onto a regular NxNxN mesh is possible, >>>>> though would be interested to hear if so. >>>>> >>>>> -d >>>>> >>>>> >>>>> On Thu, Jun 5, 2014 at 7:58 PM, Brendan Griffen >>>>> <brendan.f.griffen@gmail.com> wrote: >>>>>> >>>>>> Thanks. I'll get the "bleeding edge" version first then try >>>>>> your suggestions. Though I want to return the NxNxN array and be able to >>>>>> write this mesh to a file. It is *only* using the cic part of yt and it >>>>>> should return the mesh to be written? Just wanted to clarify? >>>>>> >>>>>> Thanks. >>>>>> Brendan >>>>>> >>>>>> >>>>>> On Thu, Jun 5, 2014 at 6:49 PM, Nathan Goldbaum >>>>>> <nathan12343@gmail.com> wrote: >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Thu, Jun 5, 2014 at 3:36 PM, John ZuHone >>>>>>> <jzuhone@gmail.com> wrote: >>>>>>>> >>>>>>>> Hi Brendan, >>>>>>>> >>>>>>>> Which version of yt are you using? >>>>>>>> >>>>>>>> If you're using 3.0, this is actually fairly easy. If you >>>>>>>> look in yt.fields.particle_fields.py, around line 85, you can see how this >>>>>>>> is done for the "particle_density" and "particle_mass" fields. Basically you >>>>>>>> can call a "deposit" method which takes the particle field quantity you want >>>>>>>> deposited and deposits it into cells. The underlying calculation is done >>>>>>>> using Cython, so it's fast. >>>>>>> >>>>>>> >>>>>>> And you shouldn't ever actually need to call these "deposit" >>>>>>> functions, since "deposit" is exposed as a field type for all datasets that >>>>>>> contain particles. >>>>>>> >>>>>>> Here is a notebook that does this for Enzo AMR data: >>>>>>> >>>>>>> >>>>>>> http://nbviewer.ipython.org/gist/ngoldbaum/5e19e4e6cc2bf330149c >>>>>>> >>>>>>> This dataset contains about a million particles and generates >>>>>>> a CIC deposition for the whole domain in about 6 seconds from a cold start. >>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> If you're using 2.x, then you can do the same thing, but it's >>>>>>>> not as straightforward. You can see how this works in >>>>>>>> yt.data_objects.universal_fields.py, around line 986, where the >>>>>>>> "particle_density" field is defined. Basically, it calls CICDeposit_3, which >>>>>>>> is in yt.utilities.lib.CICDeposit.pyx. >>>>>>>> >>>>>>>> Let me know if you need any more clarification. >>>>>>>> >>>>>>>> Best, >>>>>>>> >>>>>>>> John Z >>>>>>>> >>>>>>>> On Jun 5, 2014, at 6:07 PM, Brendan Griffen >>>>>>>> <brendan.f.griffen@gmail.com> wrote: >>>>>>>> >>>>>>>> > Hi, >>>>>>>> > >>>>>>>> > I was wondering if there were any Cython routines within yt >>>>>>>> > which takes particle data and converts it into a cloud-in-cell based mesh >>>>>>>> > which can be written to a file of my choosing. >>>>>>> >>>>>>> >>>>>>> What sort of mesh were you looking for? yt will internally >>>>>>> construct an octree if it is fed particle data. I'm not sure whether this >>>>>>> octree can be saved to disk for later analysis. >>>>>>> >>>>>>> It's also possible to create a uniform resolution covering >>>>>>> grid containing field data for a deposited quantity, which can be quite >>>>>>> easily saved to disk in a number of ways. >>>>>>> >>>>>>>> >>>>>>>> I heard a while ago there was some such functionality but it >>>>>>>> could be too far down the yt rabbit hole to be used as a standalone? Is this >>>>>>>> true? I have my own Python code for doing it but it just isn't fast enough >>>>>>>> and thought I'd ask the yt community if there were any wrapper tools >>>>>>>> available to boost the speed. >>>>>>>> > >>>>>>>> > Thanks. >>>>>>>> > Brendan >>>>>>>> > _______________________________________________ >>>>>>>> > yt-users mailing list >>>>>>>> > yt-users@lists.spacepope.org >>>>>>>> > >>>>>>>> > http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> yt-users mailing list >>>>>>>> yt-users@lists.spacepope.org >>>>>>>> >>>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >>>>>>> >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> yt-users mailing list >>>>>>> yt-users@lists.spacepope.org >>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >>>>>>> >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> yt-users mailing list >>>>>> yt-users@lists.spacepope.org >>>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >>>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> yt-users mailing list >>>>> yt-users@lists.spacepope.org >>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >>>>> >>>> >>>> >>>> _______________________________________________ >>>> yt-users mailing list >>>> yt-users@lists.spacepope.org >>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >>>> >>> >>> >>> _______________________________________________ >>> yt-users mailing list >>> yt-users@lists.spacepope.org >>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >>> >> >> >> _______________________________________________ >> yt-users mailing list >> yt-users@lists.spacepope.org >> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >> > > > _______________________________________________ > yt-users mailing list > yt-users@lists.spacepope.org > http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
Hi Matt, Thanks for your detailed email. Forgive my naivety but why do you need the oct-tree in the first place? I have a my own fortran code for constructing a cloud in cell mesh and it uses very little overhead (just the n^3 grid and the particle data itself). I then calculate the dx,dy,dzs to the nearest 8 grid points and distribute accordingly in a omp loop which is done in a fraction of a second. Does the situation with yt come about (oct tree etc.) necessarily because of the way it handles particle data? Is it essentially used to map the particles to domains in the grid or something? The machine I max memory on has 128GB and the snapshots are using 1024^3 particles. Do you have any idea of how much memory the oct-tree uses as a function of particle/grid number? I am going to try on a 256GB machine (though this is a bit of a hassle). I'll see how I go. Thanks. Brendan On Sun, Jun 8, 2014 at 6:25 PM, Matthew Turk <matthewturk@gmail.com> wrote:
Hi all,
I feel like I owe a brief explanation of why things are tricky right now, what we're planning on doing, and how we're experimenting and developing.
Presently, the particle geometry handlers build a single mesh from all particles in the dataset, along with a coarse bitmask that correlates files to regions in the domain. This requires the allocation of a single int64 array of size Nparticles, which is sorted in place and then fed into an octree construction algorithm that then spits back out the mesh. Each octree component contains 3 64-bit integers and eitehr a void pointer or a pointer to eight other octs. Increasing n_ref decreases the number of octs in this mesh; when smoothing operaitons are conducted, a second "index" mesh is created for looking up particles near mesh points. Mesh points are used for adaptive resolution smoothing and other "deposit particles on the grid somehow" operations (including SPH kernel).
Anyway, because right now it requires a global mesh to be constructed, this is expensive and requires holding a 64-bit integer in memory for each particle. I think if you're loading the particles in differently there is some additional overhead as well, but I'm still a bit surprised you OOM on a 1024^3 dataset.
In general, we don't *need* this global mesh; is can be constructed as required, which would speed up both the initial index phase as well as the final meshing process. I got about 50% of the way to implementing this last fall, but because of various concerns and deadlines I haven't finished it. I intend to get back to it probably in July, right after we put out a 3.0, so that we can have it in time for 3.1. In principle this will make the particle codes much more similar to ARTIO, in that the mesh will be constructed only as required and discarded when no longer required, which will make them much more memory efficient.
But, getting a single mesh for extremely large data is a very high priority; right now for the 10240^3 run we've been loading up individual sub-chunks, which I want to stop doing.
From the technical perspective, these are the things that need to happen on the yt side for particle datasets to move to this "lazy" mode of loading; most of this is based on things learned from 2HOT and ARTIO, and will involve converting to a forest-of-octrees.
* Split into spatially-organized subchunks of ParticleOctreeSubset objects, such that these map 1:Nfiles, and that can be constructed on the fly. * Construct a dual-mesh of the bitmask "ParticleRegion" object that will help with identifying neighbors to a given oct cell, so that if we're inside one octree we know which neighbor octrees to grab if we need particles for smoothing things (fast boundary particle identification is later down the road) * Parallel sort of particles, or using the parallel ring function; may not be necessary after all
All of this is doable, and I'd be happy to work with people if they'd like to take a shot at implementing it, but I've mostly put it on my list for post-3.0.
-Matt
On Sun, Jun 8, 2014 at 2:43 PM, Nathan Goldbaum <nathan12343@gmail.com> wrote:
On Sun, Jun 8, 2014 at 12:27 PM, Brendan Griffen <brendan.f.griffen@gmail.com> wrote:
Also, how do I construct just a zero filled yt array with dimensions (ndim,ndim,ndim)? Thanks
from yt import YTArray from numpy import np
arr = YTArray(np.zeros([ndim, ndim, ndim]), input_units=units_string)
or alternatively:
from yt.units import kiloparsec
arr = kiloparsec*np.zeros([ndim, ndim, ndim])
it doesn't have to be kiloparsec - you can compose the units you want
any of the unit symbols that live in yt.units.
See this page for a ton more detail about yt's new unit system: http://yt-project.org/docs/dev-3.0/analyzing/units/index.html
Brendan
On Sun, Jun 8, 2014 at 3:26 PM, Brendan Griffen <brendan.f.griffen@gmail.com> wrote:
Hi,
Since I get memory errors. Could I not just read in the blocks of the output individually then basically stack the mesh each time. That way
not
every single particle of the snapshot has to be loaded at the same time. Would that just be a case of doing
level = int(math.log(ndim,2)) cg = ds.covering_grid(level=level, left_edge=[0,0,0],dims=[ndim,ndim,ndim]) arr = cg['deposit', 'all_density'] arrall += arr
in a loop over each HDF5 block?
It's likely that the memory use is dominated by the octree rather than
out of the
covering grid. This is with 1024^3 particles, correct?
You can probably significantly reduce the memory used by the octree by increasing n_ref in the call to load_particles.
See this page for more detail about load_particles:
http://yt-project.org/docs/dev-3.0/examining/loading_data.html#generic-parti...
Larger n_ref means fewer octree cells (lower resolution), but it also
lower poisson noise and lower memory use.
Alternatively, as Matt suggested, you could break your 1024^3 ensemble of particles up into chunks, loop over the chunk, creating a particle octree and then a covering grid for each subset of the particles. Your final covering grid is just the sub of the covering grids for each subset of particles.
Thanks. Brendan
On Fri, Jun 6, 2014 at 7:26 PM, Matthew Turk <matthewturk@gmail.com> wrote:
On Jun 6, 2014 4:54 PM, "Brendan Griffen" <
brendan.f.griffen@gmail.com>
wrote:
OK great. It is very low resolution but it worked. Thanks for all
your
help. My higher resolution run 1024^3 in 100 Mpc seems to crash on 128GB memory machine. I might have to look elsewhere.
If you are looking for low resolution extraction you can tune the memory usage by changing the parameter n_ref to something higher.
Supporting extremely large datasets in a single mesh is on the roadmap for the late summer or fall, after a 3.0 release goes out. For now you can also extract before you load in; this is sort of how we are supporting an INCITE project with very large particle counts.
Also, I normally use Canopy distribution but I just use an alias to loadyt which erases my PYTHONPATH and I can't access scipy and a few other libraries any more. What is the best practice here? Should I just manually export PYTHONPATH and point to the libraries need in canopy or can
means they play
nice together?
Thanks.
BG
On Fri, Jun 6, 2014 at 2:54 PM, Nathan Goldbaum <nathan12343@gmail.com> wrote: > > > > > On Fri, Jun 6, 2014 at 11:48 AM, Brendan Griffen > <brendan.f.griffen@gmail.com> wrote: >> >> OK great. Thanks. I just wanted a homogeneous mesh. 512^3 with no >> nesting of any kind. Though when I plot the image it looks like it is >> assigning particles incorrectly (low resolution on the outside). This is >> just a test image. >> > > The SlicePlot is visualizing the octree so there is less resolution > where there are fewer particles. If you want to visualize the covering grid > you're going to need to visualize that separately. > >> >> ds = yt.load_particles(data, length_unit=3.08e24, >> mass_unit=1.9891e33,bbox=bbox) >> >> ad = ds.all_data() >> print ad['deposit', 'all_cic'] >> slc = yt.SlicePlot(ds, 2, ('deposit', 'all_cic')) >> slc.set_figure_size(4) >> cg = ds.covering_grid(level=9, left_edge=[0,0,0],dims=[512,512,512]) >> > > To actually produce the uniform resolution ndarray, you're going to > need to do something like: > > array = cg[('deposit', 'all_cic')] > > array will then be a 3D array you can do whatever you want with. By > default it has units, but to strip them off you'll just need to cast to > ndarray: > > array_without_units = array.v > > >> >> Also, is there a way to load multiple particle types? >> >> Do I just need to stack the particles into the array here? >> >> data = {'particle_position_x': pos[:,0], >> 'particle_position_y': pos[:,1], >> 'particle_position_z': pos[:,2], >> 'particle_mass': np.array([mpart]*npart)} >> >> Then feed it in as usual? > > > That's right, although if the particle masses are different for the > different particle types that code snippet will need to be generalized to > handle that. > > I think in principle it should be possible to make load_particles > handle different particle types just like an SPH dataset that contains > multiple particle types, but right now that hasn't been implemented yet. > >> >> >> Brendan >> >> >> On Thu, Jun 5, 2014 at 9:44 PM, Nathan Goldbaum >> <nathan12343@gmail.com> wrote: >>> >>> That's right, you can set that via the bbox keyword parameter for >>> load_particles. I'd urge you to take a look at the docstrings and source >>> code for load_particles. >>> >>> >>> On Thu, Jun 5, 2014 at 6:34 PM, Brendan Griffen >>> <brendan.f.griffen@gmail.com> wrote: >>>> >>>> Thanks very much Nathan. I tried to load in my own data but I >>>> think there are too many particles or I have to specifically set the domain >>>> size. >>>> >>>> In this area: >>>> >>>> data = {'particle_position_x': pos[:,0], >>>> 'particle_position_y': pos[:,1], >>>> 'particle_position_z': pos[:,2], >>>> 'particle_mass': np.array([mpart]*npart)} >>>> >>>> ds = yt.load_particles(data, length_unit=3.08e24, >>>> mass_unit=1.9891e36) >>>> ad = ds.all_data() >>>> print ad['deposit', 'all_cic'] >>>> >>>> In [3]: run ytcic.py >>>> yt : [INFO ] 2014-06-05 21:29:06,183 Parameters: current_time >>>> = 0.0 >>>> yt : [INFO ] 2014-06-05 21:29:06,183 Parameters: >>>> domain_dimensions = [2 2 2] >>>> yt : [INFO ] 2014-06-05 21:29:06,184 Parameters: >>>> domain_left_edge = [ 0. 0. 0.] >>>> yt : [INFO ] 2014-06-05 21:29:06,185 Parameters: >>>> domain_right_edge = [ 1. 1. 1.] >>>> yt : [INFO ] 2014-06-05 21:29:06,185 Parameters: >>>> cosmological_simulation = 0.0 >>>> yt : [INFO ] 2014-06-05 21:29:06,188 Allocating for 1.342e+08 >>>> particles >>>> >>>>
>>>> YTDomainOverflow Traceback (most recent >>>> call last) >>>> >>>> /nfs/blank/h4231/bgriffen/data/lib/yt-x86_64/lib/python2.7/site-packages/IPython/utils/py3compat.pyc >>>> in execfile(fname, *where) >>>> 202 else: >>>> 203 filename = fname >>>> --> 204 __builtin__.execfile(filename, *where) >>>> >>>> >>>> /nfs/blank/h4231/bgriffen/work/projects/caterpillar/c2ray/cic/ytcic.py in >>>> <module>() >>>> 52 >>>> 53 ad = ds.all_data() >>>> ---> 54 print ad['deposit', 'all_cic'] >>>> 55 slc = yt.SlicePlot(ds, 2, ('deposit', 'all_cic')) >>>> 56 slc.set_figure_size(4) >>>> >>>> >>>> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc >>>> in __getitem__(self, key) >>>> 205 Returns a single field. Will add if necessary. >>>> 206 """ >>>> --> 207 f = self._determine_fields([key])[0] >>>> 208 if f not in self.field_data and key not in >>>> self.field_data: >>>> 209 if f in self._container_fields: >>>> >>>> >>>> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc >>>> in _determine_fields(self, fields) >>>> 453 raise YTFieldNotParseable(field) >>>> 454 ftype, fname = field >>>> --> 455 finfo = self.pf._get_field_info(ftype, >>>> fname) >>>> 456 else: >>>> 457 fname = field >>>> >>>> >>>> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/static_output.pyc >>>> in _get_field_info(self, ftype, fname) >>>> 445 _last_finfo = None >>>> 446 def _get_field_info(self, ftype, fname = None): >>>> --> 447 self.index >>>> 448 if fname is None: >>>> 449 ftype, fname = "unknown", ftype >>>> >>>> >>>> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/static_output.pyc >>>> in index(self) >>>> 277 raise RuntimeError("You should not >>>> instantiate Dataset.") >>>> 278 self._instantiated_index = self._index_class( >>>> --> 279 self, dataset_type=self.dataset_type) >>>> 280 # Now we do things that we need an >>>> instantiated index for >>>> 281 # ...first off, we create our field_info now. >>>> >>>> >>>> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/frontends/stream/data_structures.pyc >>>> in __init__(self, pf, dataset_type) >>>> 942 def __init__(self, pf, dataset_type = None): >>>> 943 self.stream_handler = pf.stream_handler >>>> --> 944 super(StreamParticleIndex, self).__init__(pf, >>>> dataset_type) >>>> 945 >>>> 946 def _setup_data_io(self): >>>> >>>> >>>> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/geometry/particle_geometry_handler.pyc >>>> in __init__(self, pf, dataset_type) >>>> 48 self.directory = >>>> os.path.dirname(self.index_filename) >>>> 49 self.float_type = np.float64 >>>> ---> 50 super(ParticleIndex, self).__init__(pf, >>>> dataset_type) >>>> 51 >>>> 52 def _setup_geometry(self): >>>> >>>> >>>> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/geometry/geometry_handler.pyc >>>> in __init__(self, pf, dataset_type) >>>> 54 >>>> 55 mylog.debug("Setting up domain geometry.") >>>> ---> 56 self._setup_geometry() >>>> 57 >>>> 58 mylog.debug("Initializing data grid data IO") >>>> >>>> >>>> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/geometry/particle_geometry_handler.pyc >>>> in _setup_geometry(self) >>>> 52 def _setup_geometry(self): >>>> 53 mylog.debug("Initializing Particle Geometry >>>> Handler.") >>>> ---> 54 self._initialize_particle_handler() >>>> 55 >>>> 56 >>>> >>>> >>>> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/geometry/particle_geometry_handler.pyc >>>> in _initialize_particle_handler(self) >>>> 87 pf.domain_left_edge, pf.domain_right_edge, >>>> 88 [N, N, N], len(self.data_files)) >>>> ---> 89 self._initialize_indices() >>>> 90 self.oct_handler.finalize() >>>> 91 self.max_level = self.oct_handler.max_level >>>> >>>> >>>> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/geometry/particle_geometry_handler.pyc >>>> in _initialize_indices(self) >>>> 109 npart = >>>> sum(data_file.total_particles.values()) >>>> 110 morton[ind:ind + npart] = \ >>>> --> 111 self.io._initialize_index(data_file, >>>> self.regions) >>>> 112 ind += npart >>>> 113 morton.sort() >>>> >>>> >>>> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/frontends/stream/io.pyc in >>>> _initialize_index(self, data_file, regions) >>>> 144 raise YTDomainOverflow(pos.min(axis=0), >>>> pos.max(axis=0), >>>> 145 >>>> data_file.pf.domain_left_edge, >>>> --> 146 >>>> data_file.pf.domain_right_edge) >>>> 147 regions.add_data_file(pos, data_file.file_id) >>>> 148 morton.append(compute_morton( >>>> >>>> YTDomainOverflow: Particle bounds [ 0. 0. 0.] and [ 99.99999237 >>>> 99.99999237 99.99999237] exceed domain bounds [ 0. 0. 0.] code_length and >>>> [ 1. 1. 1.] code_length >>>> >>>> >>>> >>>> On Thu, Jun 5, 2014 at 8:22 PM, Nathan Goldbaum >>>> <nathan12343@gmail.com> wrote: >>>>> >>>>> Here's a worked out example that does what you're looking for >>>>> using a fake 1 million particle dataset: >>>>> >>>>> http://nbviewer.ipython.org/gist/ngoldbaum/546d37869aafe71cfe38 >>>>> >>>>> In this notebook I make use of two key yt features: >>>>> `load_particles`, and `covering_grid`. >>>>> >>>>> load_particles creates a "stream" dataset based on in-memory data >>>>> fed in as a numpy array. This dataset acts just like an on-disk simulation >>>>> dataset, but doesn't come with the baggage of needing to write a custom >>>>> frontend to read a specific data format off disk. >>>>> >>>>> covering_grid is a way to generate uniform resolution data from >>>>> an AMR dataset. It acts like a python dictionary where the keys are field >>>>> names and returns 3D numpy arrays of whatever uniform resolution you specify >>>>> when you create the covering_grid. >>>>> >>>>> Note that if you're using load_particles all of your data needs >>>>> to live in memory. If your data is too big for that you'll need to write a >>>>> frontend for your data format or use a memmap to an on-disk file somehow. >>>>> I'm not an expert on that but others on the list should be able to help out. >>>>> >>>>> Hope that gets you well on your way :) >>>>> >>>>> -Nathan >>>>> >>>>> >>>>> On Thu, Jun 5, 2014 at 5:04 PM, Desika Narayanan >>>>> <dnarayan@haverford.edu> wrote: >>>>>> >>>>>> Hey Brendan, >>>>>> >>>>>> A couple of extra tools you might find helpful in conjunction >>>>>> with Nathan's example of depositing the particles onto an octree are at: >>>>>> >>>>>> http://paste.yt-project.org/show/4737/ >>>>>> >>>>>> Where I load a gadget snapshot, and then recover the coordinates >>>>>> and width of each cell. >>>>>> >>>>>> In response to your last question - the particles are deposited >>>>>> into an octree grid (so, you'll see that the cell sizes aren't all the same >>>>>> size). I don't know if depositing onto a regular NxNxN mesh is possible, >>>>>> though would be interested to hear if so. >>>>>> >>>>>> -d >>>>>> >>>>>> >>>>>> On Thu, Jun 5, 2014 at 7:58 PM, Brendan Griffen >>>>>> <brendan.f.griffen@gmail.com> wrote: >>>>>>> >>>>>>> Thanks. I'll get the "bleeding edge" version first then try >>>>>>> your suggestions. Though I want to return the NxNxN array and be able to >>>>>>> write this mesh to a file. It is *only* using the cic part of yt and it >>>>>>> should return the mesh to be written? Just wanted to clarify? >>>>>>> >>>>>>> Thanks. >>>>>>> Brendan >>>>>>> >>>>>>> >>>>>>> On Thu, Jun 5, 2014 at 6:49 PM, Nathan Goldbaum >>>>>>> <nathan12343@gmail.com> wrote: >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Thu, Jun 5, 2014 at 3:36 PM, John ZuHone >>>>>>>> <jzuhone@gmail.com> wrote: >>>>>>>>> >>>>>>>>> Hi Brendan, >>>>>>>>> >>>>>>>>> Which version of yt are you using? >>>>>>>>> >>>>>>>>> If you're using 3.0, this is actually fairly easy. If you >>>>>>>>> look in yt.fields.particle_fields.py, around line 85, you can see how this >>>>>>>>> is done for the "particle_density" and "particle_mass" fields. Basically you >>>>>>>>> can call a "deposit" method which takes the particle field quantity you want >>>>>>>>> deposited and deposits it into cells. The underlying calculation is done >>>>>>>>> using Cython, so it's fast. >>>>>>>> >>>>>>>> >>>>>>>> And you shouldn't ever actually need to call these "deposit" >>>>>>>> functions, since "deposit" is exposed as a field type for all datasets that >>>>>>>> contain particles. >>>>>>>> >>>>>>>> Here is a notebook that does this for Enzo AMR data: >>>>>>>> >>>>>>>> >>>>>>>> http://nbviewer.ipython.org/gist/ngoldbaum/5e19e4e6cc2bf330149c >>>>>>>> >>>>>>>> This dataset contains about a million particles and generates >>>>>>>> a CIC deposition for the whole domain in about 6 seconds from a cold start. >>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> If you're using 2.x, then you can do the same thing, but it's >>>>>>>>> not as straightforward. You can see how this works in >>>>>>>>> yt.data_objects.universal_fields.py, around line 986, where the >>>>>>>>> "particle_density" field is defined. Basically, it calls CICDeposit_3, which >>>>>>>>> is in yt.utilities.lib.CICDeposit.pyx. >>>>>>>>> >>>>>>>>> Let me know if you need any more clarification. >>>>>>>>> >>>>>>>>> Best, >>>>>>>>> >>>>>>>>> John Z >>>>>>>>> >>>>>>>>> On Jun 5, 2014, at 6:07 PM, Brendan Griffen >>>>>>>>> <brendan.f.griffen@gmail.com> wrote: >>>>>>>>> >>>>>>>>> > Hi, >>>>>>>>> > >>>>>>>>> > I was wondering if there were any Cython routines within yt >>>>>>>>> > which takes particle data and converts it into a cloud-in-cell based mesh >>>>>>>>> > which can be written to a file of my choosing. >>>>>>>> >>>>>>>> >>>>>>>> What sort of mesh were you looking for? yt will internally >>>>>>>> construct an octree if it is fed particle data. I'm not sure whether this >>>>>>>> octree can be saved to disk for later analysis. >>>>>>>> >>>>>>>> It's also possible to create a uniform resolution covering >>>>>>>> grid containing field data for a deposited quantity, which can be quite >>>>>>>> easily saved to disk in a number of ways. >>>>>>>> >>>>>>>>> >>>>>>>>> I heard a while ago there was some such functionality but it >>>>>>>>> could be too far down the yt rabbit hole to be used as a standalone? Is this >>>>>>>>> true? I have my own Python code for doing it but it just isn't fast enough >>>>>>>>> and thought I'd ask the yt community if there were any wrapper tools >>>>>>>>> available to boost the speed. >>>>>>>>> > >>>>>>>>> > Thanks. >>>>>>>>> > Brendan >>>>>>>>> > _______________________________________________ >>>>>>>>> > yt-users mailing list >>>>>>>>> > yt-users@lists.spacepope.org >>>>>>>>> > >>>>>>>>> > http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> yt-users mailing list >>>>>>>>> yt-users@lists.spacepope.org >>>>>>>>> >>>>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> yt-users mailing list >>>>>>>> yt-users@lists.spacepope.org >>>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >>>>>>>> >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> yt-users mailing list >>>>>>> yt-users@lists.spacepope.org >>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >>>>>>> >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> yt-users mailing list >>>>>> yt-users@lists.spacepope.org >>>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >>>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> yt-users mailing list >>>>> yt-users@lists.spacepope.org >>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >>>>> >>>> >>>> >>>> _______________________________________________ >>>> yt-users mailing list >>>> yt-users@lists.spacepope.org >>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >>>> >>> >>> >>> _______________________________________________ >>> yt-users mailing list >>> yt-users@lists.spacepope.org >>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >>> >> >> >> _______________________________________________ >> yt-users mailing list >> yt-users@lists.spacepope.org >> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >> > > > _______________________________________________ > yt-users mailing list > yt-users@lists.spacepope.org > http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
Hi Brendan, On Sun, Jun 8, 2014 at 9:21 PM, Brendan Griffen <brendan.f.griffen@gmail.com> wrote:
Hi Matt,
Thanks for your detailed email. Forgive my naivety but why do you need the oct-tree in the first place? I have a my own fortran code for constructing a cloud in cell mesh and it uses very little overhead (just the n^3 grid and the particle data itself). I then calculate the dx,dy,dzs to the nearest 8 grid points and distribute accordingly in a omp loop which is done in a fraction of a second. Does the situation with yt come about (oct tree etc.) necessarily because of the way it handles particle data? Is it essentially used to map the particles to domains in the grid or something?
That's not naive at all. There are two reasons -- 1) The octree is used for indexing for neighbor lookups and early-termination of region selection for particles 2) The octree is used to estimate the "required resolution" for any operation that requires a space-filling value. (i.e., any time that a particle becomes a volume.) Projections in yt are adaptive, in that they project down to the finest appropriate resolution. There's also the "arbitrary_grid" operation, which does precisely what you're describing, but as it stands right now the octree gets constructed at time of instantiation of the indexing system. Thinking it over, you may be able to avoid that completely by not using load_particles and instead using load_uniform_grid and supplying your desired dimensions. The field names should be the same.
The machine I max memory on has 128GB and the snapshots are using 1024^3 particles. Do you have any idea of how much memory the oct-tree uses as a function of particle/grid number? I am going to try on a 256GB machine (though this is a bit of a hassle). I'll see how I go.
I am disappointed that it's blowing out your RAM. This week I will try to get some memory profiling done. Could you file a bug to this effect, which will help me track it? Peak memory usage during indexing should only be 64 bits * Nparticles, unless you're using load_particles, in which case all the fields will *also* have to be in memory. It's about 8 gigabytes per field. So, I think there's something going wrong.
Thanks.
Brendan
On Sun, Jun 8, 2014 at 6:25 PM, Matthew Turk <matthewturk@gmail.com> wrote:
Hi all,
I feel like I owe a brief explanation of why things are tricky right now, what we're planning on doing, and how we're experimenting and developing.
Presently, the particle geometry handlers build a single mesh from all particles in the dataset, along with a coarse bitmask that correlates files to regions in the domain. This requires the allocation of a single int64 array of size Nparticles, which is sorted in place and then fed into an octree construction algorithm that then spits back out the mesh. Each octree component contains 3 64-bit integers and eitehr a void pointer or a pointer to eight other octs. Increasing n_ref decreases the number of octs in this mesh; when smoothing operaitons are conducted, a second "index" mesh is created for looking up particles near mesh points. Mesh points are used for adaptive resolution smoothing and other "deposit particles on the grid somehow" operations (including SPH kernel).
Anyway, because right now it requires a global mesh to be constructed, this is expensive and requires holding a 64-bit integer in memory for each particle. I think if you're loading the particles in differently there is some additional overhead as well, but I'm still a bit surprised you OOM on a 1024^3 dataset.
In general, we don't *need* this global mesh; is can be constructed as required, which would speed up both the initial index phase as well as the final meshing process. I got about 50% of the way to implementing this last fall, but because of various concerns and deadlines I haven't finished it. I intend to get back to it probably in July, right after we put out a 3.0, so that we can have it in time for 3.1. In principle this will make the particle codes much more similar to ARTIO, in that the mesh will be constructed only as required and discarded when no longer required, which will make them much more memory efficient.
But, getting a single mesh for extremely large data is a very high priority; right now for the 10240^3 run we've been loading up individual sub-chunks, which I want to stop doing.
From the technical perspective, these are the things that need to happen on the yt side for particle datasets to move to this "lazy" mode of loading; most of this is based on things learned from 2HOT and ARTIO, and will involve converting to a forest-of-octrees.
* Split into spatially-organized subchunks of ParticleOctreeSubset objects, such that these map 1:Nfiles, and that can be constructed on the fly. * Construct a dual-mesh of the bitmask "ParticleRegion" object that will help with identifying neighbors to a given oct cell, so that if we're inside one octree we know which neighbor octrees to grab if we need particles for smoothing things (fast boundary particle identification is later down the road) * Parallel sort of particles, or using the parallel ring function; may not be necessary after all
All of this is doable, and I'd be happy to work with people if they'd like to take a shot at implementing it, but I've mostly put it on my list for post-3.0.
-Matt
On Sun, Jun 8, 2014 at 2:43 PM, Nathan Goldbaum <nathan12343@gmail.com> wrote:
On Sun, Jun 8, 2014 at 12:27 PM, Brendan Griffen <brendan.f.griffen@gmail.com> wrote:
Also, how do I construct just a zero filled yt array with dimensions (ndim,ndim,ndim)? Thanks
from yt import YTArray from numpy import np
arr = YTArray(np.zeros([ndim, ndim, ndim]), input_units=units_string)
or alternatively:
from yt.units import kiloparsec
arr = kiloparsec*np.zeros([ndim, ndim, ndim])
it doesn't have to be kiloparsec - you can compose the units you want out of any of the unit symbols that live in yt.units.
See this page for a ton more detail about yt's new unit system: http://yt-project.org/docs/dev-3.0/analyzing/units/index.html
Brendan
On Sun, Jun 8, 2014 at 3:26 PM, Brendan Griffen <brendan.f.griffen@gmail.com> wrote:
Hi,
Since I get memory errors. Could I not just read in the blocks of the output individually then basically stack the mesh each time. That way not every single particle of the snapshot has to be loaded at the same time. Would that just be a case of doing
level = int(math.log(ndim,2)) cg = ds.covering_grid(level=level, left_edge=[0,0,0],dims=[ndim,ndim,ndim]) arr = cg['deposit', 'all_density'] arrall += arr
in a loop over each HDF5 block?
It's likely that the memory use is dominated by the octree rather than the covering grid. This is with 1024^3 particles, correct?
You can probably significantly reduce the memory used by the octree by increasing n_ref in the call to load_particles.
See this page for more detail about load_particles:
http://yt-project.org/docs/dev-3.0/examining/loading_data.html#generic-parti...
Larger n_ref means fewer octree cells (lower resolution), but it also means lower poisson noise and lower memory use.
Alternatively, as Matt suggested, you could break your 1024^3 ensemble of particles up into chunks, loop over the chunk, creating a particle octree and then a covering grid for each subset of the particles. Your final covering grid is just the sub of the covering grids for each subset of particles.
Thanks. Brendan
On Fri, Jun 6, 2014 at 7:26 PM, Matthew Turk <matthewturk@gmail.com> wrote:
On Jun 6, 2014 4:54 PM, "Brendan Griffen" <brendan.f.griffen@gmail.com> wrote: > > OK great. It is very low resolution but it worked. Thanks for all > your > help. My higher resolution run 1024^3 in 100 Mpc seems to crash on > 128GB > memory machine. I might have to look elsewhere. >
If you are looking for low resolution extraction you can tune the memory usage by changing the parameter n_ref to something higher.
Supporting extremely large datasets in a single mesh is on the roadmap for the late summer or fall, after a 3.0 release goes out. For now you can also extract before you load in; this is sort of how we are supporting an INCITE project with very large particle counts.
> Also, I normally use Canopy distribution but I just use an alias to > loadyt which erases my PYTHONPATH and I can't access scipy and a > few other > libraries any more. What is the best practice here? Should I just > manually > export PYTHONPATH and point to the libraries need in canopy or can > they play > nice together? > > Thanks. > > BG > > > On Fri, Jun 6, 2014 at 2:54 PM, Nathan Goldbaum > <nathan12343@gmail.com> wrote: >> >> >> >> >> On Fri, Jun 6, 2014 at 11:48 AM, Brendan Griffen >> <brendan.f.griffen@gmail.com> wrote: >>> >>> OK great. Thanks. I just wanted a homogeneous mesh. 512^3 with no >>> nesting of any kind. Though when I plot the image it looks like >>> it is >>> assigning particles incorrectly (low resolution on the outside). >>> This is >>> just a test image. >>> >> >> The SlicePlot is visualizing the octree so there is less >> resolution >> where there are fewer particles. If you want to visualize the >> covering grid >> you're going to need to visualize that separately. >> >>> >>> ds = yt.load_particles(data, length_unit=3.08e24, >>> mass_unit=1.9891e33,bbox=bbox) >>> >>> ad = ds.all_data() >>> print ad['deposit', 'all_cic'] >>> slc = yt.SlicePlot(ds, 2, ('deposit', 'all_cic')) >>> slc.set_figure_size(4) >>> cg = ds.covering_grid(level=9, >>> left_edge=[0,0,0],dims=[512,512,512]) >>> >> >> To actually produce the uniform resolution ndarray, you're going >> to >> need to do something like: >> >> array = cg[('deposit', 'all_cic')] >> >> array will then be a 3D array you can do whatever you want with. >> By >> default it has units, but to strip them off you'll just need to >> cast to >> ndarray: >> >> array_without_units = array.v >> >> >>> >>> Also, is there a way to load multiple particle types? >>> >>> Do I just need to stack the particles into the array here? >>> >>> data = {'particle_position_x': pos[:,0], >>> 'particle_position_y': pos[:,1], >>> 'particle_position_z': pos[:,2], >>> 'particle_mass': np.array([mpart]*npart)} >>> >>> Then feed it in as usual? >> >> >> That's right, although if the particle masses are different for >> the >> different particle types that code snippet will need to be >> generalized to >> handle that. >> >> I think in principle it should be possible to make load_particles >> handle different particle types just like an SPH dataset that >> contains >> multiple particle types, but right now that hasn't been >> implemented yet. >> >>> >>> >>> Brendan >>> >>> >>> On Thu, Jun 5, 2014 at 9:44 PM, Nathan Goldbaum >>> <nathan12343@gmail.com> wrote: >>>> >>>> That's right, you can set that via the bbox keyword parameter >>>> for >>>> load_particles. I'd urge you to take a look at the docstrings >>>> and source >>>> code for load_particles. >>>> >>>> >>>> On Thu, Jun 5, 2014 at 6:34 PM, Brendan Griffen >>>> <brendan.f.griffen@gmail.com> wrote: >>>>> >>>>> Thanks very much Nathan. I tried to load in my own data but I >>>>> think there are too many particles or I have to specifically >>>>> set the domain >>>>> size. >>>>> >>>>> In this area: >>>>> >>>>> data = {'particle_position_x': pos[:,0], >>>>> 'particle_position_y': pos[:,1], >>>>> 'particle_position_z': pos[:,2], >>>>> 'particle_mass': np.array([mpart]*npart)} >>>>> >>>>> ds = yt.load_particles(data, length_unit=3.08e24, >>>>> mass_unit=1.9891e36) >>>>> ad = ds.all_data() >>>>> print ad['deposit', 'all_cic'] >>>>> >>>>> In [3]: run ytcic.py >>>>> yt : [INFO ] 2014-06-05 21:29:06,183 Parameters: >>>>> current_time >>>>> = 0.0 >>>>> yt : [INFO ] 2014-06-05 21:29:06,183 Parameters: >>>>> domain_dimensions = [2 2 2] >>>>> yt : [INFO ] 2014-06-05 21:29:06,184 Parameters: >>>>> domain_left_edge = [ 0. 0. 0.] >>>>> yt : [INFO ] 2014-06-05 21:29:06,185 Parameters: >>>>> domain_right_edge = [ 1. 1. 1.] >>>>> yt : [INFO ] 2014-06-05 21:29:06,185 Parameters: >>>>> cosmological_simulation = 0.0 >>>>> yt : [INFO ] 2014-06-05 21:29:06,188 Allocating for >>>>> 1.342e+08 >>>>> particles >>>>> >>>>> >>>>> --------------------------------------------------------------------------- >>>>> YTDomainOverflow Traceback (most >>>>> recent >>>>> call last) >>>>> >>>>> >>>>> /nfs/blank/h4231/bgriffen/data/lib/yt-x86_64/lib/python2.7/site-packages/IPython/utils/py3compat.pyc >>>>> in execfile(fname, *where) >>>>> 202 else: >>>>> 203 filename = fname >>>>> --> 204 __builtin__.execfile(filename, *where) >>>>> >>>>> >>>>> >>>>> /nfs/blank/h4231/bgriffen/work/projects/caterpillar/c2ray/cic/ytcic.py in >>>>> <module>() >>>>> 52 >>>>> 53 ad = ds.all_data() >>>>> ---> 54 print ad['deposit', 'all_cic'] >>>>> 55 slc = yt.SlicePlot(ds, 2, ('deposit', 'all_cic')) >>>>> 56 slc.set_figure_size(4) >>>>> >>>>> >>>>> >>>>> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc >>>>> in __getitem__(self, key) >>>>> 205 Returns a single field. Will add if necessary. >>>>> 206 """ >>>>> --> 207 f = self._determine_fields([key])[0] >>>>> 208 if f not in self.field_data and key not in >>>>> self.field_data: >>>>> 209 if f in self._container_fields: >>>>> >>>>> >>>>> >>>>> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc >>>>> in _determine_fields(self, fields) >>>>> 453 raise YTFieldNotParseable(field) >>>>> 454 ftype, fname = field >>>>> --> 455 finfo = self.pf._get_field_info(ftype, >>>>> fname) >>>>> 456 else: >>>>> 457 fname = field >>>>> >>>>> >>>>> >>>>> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/static_output.pyc >>>>> in _get_field_info(self, ftype, fname) >>>>> 445 _last_finfo = None >>>>> 446 def _get_field_info(self, ftype, fname = None): >>>>> --> 447 self.index >>>>> 448 if fname is None: >>>>> 449 ftype, fname = "unknown", ftype >>>>> >>>>> >>>>> >>>>> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/static_output.pyc >>>>> in index(self) >>>>> 277 raise RuntimeError("You should not >>>>> instantiate Dataset.") >>>>> 278 self._instantiated_index = >>>>> self._index_class( >>>>> --> 279 self, dataset_type=self.dataset_type) >>>>> 280 # Now we do things that we need an >>>>> instantiated index for >>>>> 281 # ...first off, we create our field_info >>>>> now. >>>>> >>>>> >>>>> >>>>> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/frontends/stream/data_structures.pyc >>>>> in __init__(self, pf, dataset_type) >>>>> 942 def __init__(self, pf, dataset_type = None): >>>>> 943 self.stream_handler = pf.stream_handler >>>>> --> 944 super(StreamParticleIndex, self).__init__(pf, >>>>> dataset_type) >>>>> 945 >>>>> 946 def _setup_data_io(self): >>>>> >>>>> >>>>> >>>>> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/geometry/particle_geometry_handler.pyc >>>>> in __init__(self, pf, dataset_type) >>>>> 48 self.directory = >>>>> os.path.dirname(self.index_filename) >>>>> 49 self.float_type = np.float64 >>>>> ---> 50 super(ParticleIndex, self).__init__(pf, >>>>> dataset_type) >>>>> 51 >>>>> 52 def _setup_geometry(self): >>>>> >>>>> >>>>> >>>>> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/geometry/geometry_handler.pyc >>>>> in __init__(self, pf, dataset_type) >>>>> 54 >>>>> 55 mylog.debug("Setting up domain geometry.") >>>>> ---> 56 self._setup_geometry() >>>>> 57 >>>>> 58 mylog.debug("Initializing data grid data IO") >>>>> >>>>> >>>>> >>>>> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/geometry/particle_geometry_handler.pyc >>>>> in _setup_geometry(self) >>>>> 52 def _setup_geometry(self): >>>>> 53 mylog.debug("Initializing Particle Geometry >>>>> Handler.") >>>>> ---> 54 self._initialize_particle_handler() >>>>> 55 >>>>> 56 >>>>> >>>>> >>>>> >>>>> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/geometry/particle_geometry_handler.pyc >>>>> in _initialize_particle_handler(self) >>>>> 87 pf.domain_left_edge, >>>>> pf.domain_right_edge, >>>>> 88 [N, N, N], len(self.data_files)) >>>>> ---> 89 self._initialize_indices() >>>>> 90 self.oct_handler.finalize() >>>>> 91 self.max_level = self.oct_handler.max_level >>>>> >>>>> >>>>> >>>>> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/geometry/particle_geometry_handler.pyc >>>>> in _initialize_indices(self) >>>>> 109 npart = >>>>> sum(data_file.total_particles.values()) >>>>> 110 morton[ind:ind + npart] = \ >>>>> --> 111 self.io._initialize_index(data_file, >>>>> self.regions) >>>>> 112 ind += npart >>>>> 113 morton.sort() >>>>> >>>>> >>>>> >>>>> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/frontends/stream/io.pyc in >>>>> _initialize_index(self, data_file, regions) >>>>> 144 raise YTDomainOverflow(pos.min(axis=0), >>>>> pos.max(axis=0), >>>>> 145 >>>>> data_file.pf.domain_left_edge, >>>>> --> 146 >>>>> data_file.pf.domain_right_edge) >>>>> 147 regions.add_data_file(pos, >>>>> data_file.file_id) >>>>> 148 morton.append(compute_morton( >>>>> >>>>> YTDomainOverflow: Particle bounds [ 0. 0. 0.] and [ >>>>> 99.99999237 >>>>> 99.99999237 99.99999237] exceed domain bounds [ 0. 0. 0.] >>>>> code_length and >>>>> [ 1. 1. 1.] code_length >>>>> >>>>> >>>>> >>>>> On Thu, Jun 5, 2014 at 8:22 PM, Nathan Goldbaum >>>>> <nathan12343@gmail.com> wrote: >>>>>> >>>>>> Here's a worked out example that does what you're looking for >>>>>> using a fake 1 million particle dataset: >>>>>> >>>>>> >>>>>> http://nbviewer.ipython.org/gist/ngoldbaum/546d37869aafe71cfe38 >>>>>> >>>>>> In this notebook I make use of two key yt features: >>>>>> `load_particles`, and `covering_grid`. >>>>>> >>>>>> load_particles creates a "stream" dataset based on in-memory >>>>>> data >>>>>> fed in as a numpy array. This dataset acts just like an >>>>>> on-disk simulation >>>>>> dataset, but doesn't come with the baggage of needing to write >>>>>> a custom >>>>>> frontend to read a specific data format off disk. >>>>>> >>>>>> covering_grid is a way to generate uniform resolution data >>>>>> from >>>>>> an AMR dataset. It acts like a python dictionary where the >>>>>> keys are field >>>>>> names and returns 3D numpy arrays of whatever uniform >>>>>> resolution you specify >>>>>> when you create the covering_grid. >>>>>> >>>>>> Note that if you're using load_particles all of your data >>>>>> needs >>>>>> to live in memory. If your data is too big for that you'll >>>>>> need to write a >>>>>> frontend for your data format or use a memmap to an on-disk >>>>>> file somehow. >>>>>> I'm not an expert on that but others on the list should be >>>>>> able to help out. >>>>>> >>>>>> Hope that gets you well on your way :) >>>>>> >>>>>> -Nathan >>>>>> >>>>>> >>>>>> On Thu, Jun 5, 2014 at 5:04 PM, Desika Narayanan >>>>>> <dnarayan@haverford.edu> wrote: >>>>>>> >>>>>>> Hey Brendan, >>>>>>> >>>>>>> A couple of extra tools you might find helpful in conjunction >>>>>>> with Nathan's example of depositing the particles onto an >>>>>>> octree are at: >>>>>>> >>>>>>> http://paste.yt-project.org/show/4737/ >>>>>>> >>>>>>> Where I load a gadget snapshot, and then recover the >>>>>>> coordinates >>>>>>> and width of each cell. >>>>>>> >>>>>>> In response to your last question - the particles are >>>>>>> deposited >>>>>>> into an octree grid (so, you'll see that the cell sizes >>>>>>> aren't all the same >>>>>>> size). I don't know if depositing onto a regular NxNxN mesh >>>>>>> is possible, >>>>>>> though would be interested to hear if so. >>>>>>> >>>>>>> -d >>>>>>> >>>>>>> >>>>>>> On Thu, Jun 5, 2014 at 7:58 PM, Brendan Griffen >>>>>>> <brendan.f.griffen@gmail.com> wrote: >>>>>>>> >>>>>>>> Thanks. I'll get the "bleeding edge" version first then try >>>>>>>> your suggestions. Though I want to return the NxNxN array >>>>>>>> and be able to >>>>>>>> write this mesh to a file. It is *only* using the cic part >>>>>>>> of yt and it >>>>>>>> should return the mesh to be written? Just wanted to >>>>>>>> clarify? >>>>>>>> >>>>>>>> Thanks. >>>>>>>> Brendan >>>>>>>> >>>>>>>> >>>>>>>> On Thu, Jun 5, 2014 at 6:49 PM, Nathan Goldbaum >>>>>>>> <nathan12343@gmail.com> wrote: >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Thu, Jun 5, 2014 at 3:36 PM, John ZuHone >>>>>>>>> <jzuhone@gmail.com> wrote: >>>>>>>>>> >>>>>>>>>> Hi Brendan, >>>>>>>>>> >>>>>>>>>> Which version of yt are you using? >>>>>>>>>> >>>>>>>>>> If you're using 3.0, this is actually fairly easy. If you >>>>>>>>>> look in yt.fields.particle_fields.py, around line 85, you >>>>>>>>>> can see how this >>>>>>>>>> is done for the "particle_density" and "particle_mass" >>>>>>>>>> fields. Basically you >>>>>>>>>> can call a "deposit" method which takes the particle field >>>>>>>>>> quantity you want >>>>>>>>>> deposited and deposits it into cells. The underlying >>>>>>>>>> calculation is done >>>>>>>>>> using Cython, so it's fast. >>>>>>>>> >>>>>>>>> >>>>>>>>> And you shouldn't ever actually need to call these >>>>>>>>> "deposit" >>>>>>>>> functions, since "deposit" is exposed as a field type for >>>>>>>>> all datasets that >>>>>>>>> contain particles. >>>>>>>>> >>>>>>>>> Here is a notebook that does this for Enzo AMR data: >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> http://nbviewer.ipython.org/gist/ngoldbaum/5e19e4e6cc2bf330149c >>>>>>>>> >>>>>>>>> This dataset contains about a million particles and >>>>>>>>> generates >>>>>>>>> a CIC deposition for the whole domain in about 6 seconds >>>>>>>>> from a cold start. >>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> If you're using 2.x, then you can do the same thing, but >>>>>>>>>> it's >>>>>>>>>> not as straightforward. You can see how this works in >>>>>>>>>> yt.data_objects.universal_fields.py, around line 986, >>>>>>>>>> where the >>>>>>>>>> "particle_density" field is defined. Basically, it calls >>>>>>>>>> CICDeposit_3, which >>>>>>>>>> is in yt.utilities.lib.CICDeposit.pyx. >>>>>>>>>> >>>>>>>>>> Let me know if you need any more clarification. >>>>>>>>>> >>>>>>>>>> Best, >>>>>>>>>> >>>>>>>>>> John Z >>>>>>>>>> >>>>>>>>>> On Jun 5, 2014, at 6:07 PM, Brendan Griffen >>>>>>>>>> <brendan.f.griffen@gmail.com> wrote: >>>>>>>>>> >>>>>>>>>> > Hi, >>>>>>>>>> > >>>>>>>>>> > I was wondering if there were any Cython routines within >>>>>>>>>> > yt >>>>>>>>>> > which takes particle data and converts it into a >>>>>>>>>> > cloud-in-cell based mesh >>>>>>>>>> > which can be written to a file of my choosing. >>>>>>>>> >>>>>>>>> >>>>>>>>> What sort of mesh were you looking for? yt will internally >>>>>>>>> construct an octree if it is fed particle data. I'm not >>>>>>>>> sure whether this >>>>>>>>> octree can be saved to disk for later analysis. >>>>>>>>> >>>>>>>>> It's also possible to create a uniform resolution covering >>>>>>>>> grid containing field data for a deposited quantity, which >>>>>>>>> can be quite >>>>>>>>> easily saved to disk in a number of ways. >>>>>>>>> >>>>>>>>>> >>>>>>>>>> I heard a while ago there was some such functionality but >>>>>>>>>> it >>>>>>>>>> could be too far down the yt rabbit hole to be used as a >>>>>>>>>> standalone? Is this >>>>>>>>>> true? I have my own Python code for doing it but it just >>>>>>>>>> isn't fast enough >>>>>>>>>> and thought I'd ask the yt community if there were any >>>>>>>>>> wrapper tools >>>>>>>>>> available to boost the speed. >>>>>>>>>> > >>>>>>>>>> > Thanks. >>>>>>>>>> > Brendan >>>>>>>>>> > _______________________________________________ >>>>>>>>>> > yt-users mailing list >>>>>>>>>> > yt-users@lists.spacepope.org >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> yt-users mailing list >>>>>>>>>> yt-users@lists.spacepope.org >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> yt-users mailing list >>>>>>>>> yt-users@lists.spacepope.org >>>>>>>>> >>>>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> yt-users mailing list >>>>>>>> yt-users@lists.spacepope.org >>>>>>>> >>>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >>>>>>>> >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> yt-users mailing list >>>>>>> yt-users@lists.spacepope.org >>>>>>> >>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >>>>>>> >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> yt-users mailing list >>>>>> yt-users@lists.spacepope.org >>>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >>>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> yt-users mailing list >>>>> yt-users@lists.spacepope.org >>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >>>>> >>>> >>>> >>>> _______________________________________________ >>>> yt-users mailing list >>>> yt-users@lists.spacepope.org >>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >>>> >>> >>> >>> _______________________________________________ >>> yt-users mailing list >>> yt-users@lists.spacepope.org >>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >>> >> >> >> _______________________________________________ >> yt-users mailing list >> yt-users@lists.spacepope.org >> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >> > > > _______________________________________________ > yt-users mailing list > yt-users@lists.spacepope.org > http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
This is the full error if it helps at all? It is indeed, loading in all of the quantities. Loading particles... --> Loading particle type: 1 yt : [INFO ] 2014-06-08 15:12:05,540 Parameters: current_time = 0.0 yt : [INFO ] 2014-06-08 15:12:05,540 Parameters: domain_dimensions = [2 2 2] yt : [INFO ] 2014-06-08 15:12:05,541 Parameters: domain_left_edge = [ 0. 0. 0.] yt : [INFO ] 2014-06-08 15:12:05,542 Parameters: domain_right_edge = [ 100. 100. 100.] yt : [INFO ] 2014-06-08 15:12:05,542 Parameters: cosmological_simulation = 0.0 yt : [INFO ] 2014-06-08 15:12:05,548 Allocating for 1.074e+09 particles yt : [INFO ] 2014-06-08 15:16:16,195 Identified 7.584e+07 octs yt : [INFO ] 2014-06-08 15:16:16,299 Loading field plugins. yt : [INFO ] 2014-06-08 15:16:16,299 Loaded angular_momentum (8 new fields) yt : [INFO ] 2014-06-08 15:16:16,299 Loaded astro (14 new fields) yt : [INFO ] 2014-06-08 15:16:16,300 Loaded cosmology (20 new fields) yt : [INFO ] 2014-06-08 15:16:16,300 Loaded fluid (56 new fields) yt : [INFO ] 2014-06-08 15:16:16,301 Loaded fluid_vector (88 new fields) yt : [INFO ] 2014-06-08 15:16:16,301 Loaded geometric (103 new fields) yt : [INFO ] 2014-06-08 15:16:16,301 Loaded local (103 new fields) yt : [INFO ] 2014-06-08 15:16:16,302 Loaded magnetic_field (109 new fields) yt : [INFO ] 2014-06-08 15:16:16,302 Loaded species (109 new fields) --------------------------------------------------------------------------- MemoryError Traceback (most recent call last) /nfs/blank/h4231/bgriffen/data/lib/yt-x86_64/lib/python2.7/site-packages/IPython/utils/py3compat.pyc in execfile(fname, *where) 202 else: 203 filename = fname --> 204 __builtin__.execfile(filename, *where) /nfs/blank/h4231/bgriffen/work/projects/caterpillar/c2ray/cic/ytcic.py in <module>() 99 slc.set_figure_size(4) 100 slc.save() --> 101 102 for ndim in dimlist: 103 print /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc in __getitem__(self, key) 218 return self.field_data[f] 219 else: --> 220 self.get_data(f) 221 # fi.units is the unit expression string. We depend on the registry 222 # hanging off the dataset to define this unit object. /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc in get_data(self, fields) 627 628 fields_to_generate += gen_fluids + gen_particles --> 629 self._generate_fields(fields_to_generate) 630 631 def _generate_fields(self, fields_to_generate): /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc in _generate_fields(self, fields_to_generate) 644 fi = self.pf._get_field_info(*field) 645 try: --> 646 fd = self._generate_field(field) 647 if type(fd) == np.ndarray: 648 fd = self.pf.arr(fd, fi.units) /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc in _generate_field(self, field) 255 tr = self._generate_particle_field(field) 256 else: --> 257 tr = self._generate_fluid_field(field) 258 if tr is None: 259 raise YTCouldNotGenerateField(field, self.pf) /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc in _generate_fluid_field(self, field) 273 finfo.check_available(gen_obj) 274 except NeedsGridType as ngt_exception: --> 275 rv = self._generate_spatial_fluid(field, ngt_exception.ghost_zones) 276 else: 277 rv = finfo(gen_obj) /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc in _generate_spatial_fluid(self, field, ngz) 289 o = self._current_chunk.objs[0] 290 with o._activate_cache(): --> 291 ind += o.select(self.selector, self[field], rv, ind) 292 else: 293 chunks = self.index._chunk(self, "spatial", ngz = ngz) /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc in __getitem__(self, key) 218 return self.field_data[f] 219 else: --> 220 self.get_data(f) 221 # fi.units is the unit expression string. We depend on the registry 222 # hanging off the dataset to define this unit object. /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc in get_data(self, fields) 627 628 fields_to_generate += gen_fluids + gen_particles --> 629 self._generate_fields(fields_to_generate) 630 631 def _generate_fields(self, fields_to_generate): /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc in _generate_fields(self, fields_to_generate) 644 fi = self.pf._get_field_info(*field) 645 try: --> 646 fd = self._generate_field(field) 647 if type(fd) == np.ndarray: 648 fd = self.pf.arr(fd, fi.units) /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc in _generate_field(self, field) 255 tr = self._generate_particle_field(field) 256 else: --> 257 tr = self._generate_fluid_field(field) 258 if tr is None: 259 raise YTCouldNotGenerateField(field, self.pf) /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc in _generate_fluid_field(self, field) 275 rv = self._generate_spatial_fluid(field, ngt_exception.ghost_zones) 276 else: --> 277 rv = finfo(gen_obj) 278 return rv 279 /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/fields/derived_field.pyc in __call__(self, data) 176 "for %s" % (self.name,)) 177 with self.unit_registry(data): --> 178 dd = self._function(self, data) 179 for field_name in data.keys(): 180 if field_name not in original_fields: /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/fields/particle_fields.pyc in particle_cic(field, data) 113 def particle_cic(field, data): 114 pos = data[ptype, coord_name] --> 115 d = data.deposit(pos, [data[ptype, mass_name]], method = "cic") 116 d = data.apply_units(d, data[ptype, mass_name].units) 117 d /= data["index", "cell_volume"] /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/octree_subset.pyc in deposit(self, positions, fields, method) 167 mylog.debug("Depositing %s (%s^3) particles into %s Octs", 168 positions.shape[0], positions.shape[0]**0.3333333, nvals[-1]) --> 169 pos = np.asarray(positions.convert_to_units("code_length"), 170 dtype="float64") 171 # We should not need the following if we know in advance all our fields /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/units/yt_array.pyc in convert_to_units(self, units) 366 367 self.units = new_units --> 368 self *= conversion_factor 369 return self 370 /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/units/yt_array.pyc in __imul__(self, other) 667 """ See __mul__. """ 668 oth = sanitize_units_mul(self, other) --> 669 return np.multiply(self, oth, out=self) 670 671 def __div__(self, right_object): /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/units/yt_array.pyc in __array_wrap__(self, out_arr, context) 966 # casting to YTArray avoids creating a YTQuantity with size > 1 967 return YTArray(np.array(out_arr, unit)) --> 968 return ret_class(np.array(out_arr), unit) 969 970 MemoryError: On Sun, Jun 8, 2014 at 10:50 PM, Matthew Turk <matthewturk@gmail.com> wrote:
Hi Brendan,
Hi Matt,
Thanks for your detailed email. Forgive my naivety but why do you need
On Sun, Jun 8, 2014 at 9:21 PM, Brendan Griffen <brendan.f.griffen@gmail.com> wrote: the
oct-tree in the first place? I have a my own fortran code for constructing a cloud in cell mesh and it uses very little overhead (just the n^3 grid and the particle data itself). I then calculate the dx,dy,dzs to the nearest 8 grid points and distribute accordingly in a omp loop which is done in a fraction of a second. Does the situation with yt come about (oct tree etc.) necessarily because of the way it handles particle data? Is it essentially used to map the particles to domains in the grid or something?
That's not naive at all. There are two reasons --
1) The octree is used for indexing for neighbor lookups and early-termination of region selection for particles 2) The octree is used to estimate the "required resolution" for any operation that requires a space-filling value. (i.e., any time that a particle becomes a volume.)
Projections in yt are adaptive, in that they project down to the finest appropriate resolution. There's also the "arbitrary_grid" operation, which does precisely what you're describing, but as it stands right now the octree gets constructed at time of instantiation of the indexing system. Thinking it over, you may be able to avoid that completely by not using load_particles and instead using load_uniform_grid and supplying your desired dimensions. The field names should be the same.
The machine I max memory on has 128GB and the snapshots are using 1024^3 particles. Do you have any idea of how much memory the oct-tree uses as a function of particle/grid number? I am going to try on a 256GB machine (though this is a bit of a hassle). I'll see how I go.
I am disappointed that it's blowing out your RAM. This week I will try to get some memory profiling done. Could you file a bug to this effect, which will help me track it? Peak memory usage during indexing should only be 64 bits * Nparticles, unless you're using load_particles, in which case all the fields will *also* have to be in memory. It's about 8 gigabytes per field. So, I think there's something going wrong.
Thanks.
Brendan
On Sun, Jun 8, 2014 at 6:25 PM, Matthew Turk <matthewturk@gmail.com>
Hi all,
I feel like I owe a brief explanation of why things are tricky right now, what we're planning on doing, and how we're experimenting and developing.
Presently, the particle geometry handlers build a single mesh from all particles in the dataset, along with a coarse bitmask that correlates files to regions in the domain. This requires the allocation of a single int64 array of size Nparticles, which is sorted in place and then fed into an octree construction algorithm that then spits back out the mesh. Each octree component contains 3 64-bit integers and eitehr a void pointer or a pointer to eight other octs. Increasing n_ref decreases the number of octs in this mesh; when smoothing operaitons are conducted, a second "index" mesh is created for looking up particles near mesh points. Mesh points are used for adaptive resolution smoothing and other "deposit particles on the grid somehow" operations (including SPH kernel).
Anyway, because right now it requires a global mesh to be constructed, this is expensive and requires holding a 64-bit integer in memory for each particle. I think if you're loading the particles in differently there is some additional overhead as well, but I'm still a bit surprised you OOM on a 1024^3 dataset.
In general, we don't *need* this global mesh; is can be constructed as required, which would speed up both the initial index phase as well as the final meshing process. I got about 50% of the way to implementing this last fall, but because of various concerns and deadlines I haven't finished it. I intend to get back to it probably in July, right after we put out a 3.0, so that we can have it in time for 3.1. In principle this will make the particle codes much more similar to ARTIO, in that the mesh will be constructed only as required and discarded when no longer required, which will make them much more memory efficient.
But, getting a single mesh for extremely large data is a very high priority; right now for the 10240^3 run we've been loading up individual sub-chunks, which I want to stop doing.
From the technical perspective, these are the things that need to happen on the yt side for particle datasets to move to this "lazy" mode of loading; most of this is based on things learned from 2HOT and ARTIO, and will involve converting to a forest-of-octrees.
* Split into spatially-organized subchunks of ParticleOctreeSubset objects, such that these map 1:Nfiles, and that can be constructed on the fly. * Construct a dual-mesh of the bitmask "ParticleRegion" object that will help with identifying neighbors to a given oct cell, so that if we're inside one octree we know which neighbor octrees to grab if we need particles for smoothing things (fast boundary particle identification is later down the road) * Parallel sort of particles, or using the parallel ring function; may not be necessary after all
All of this is doable, and I'd be happy to work with people if they'd like to take a shot at implementing it, but I've mostly put it on my list for post-3.0.
-Matt
On Sun, Jun 8, 2014 at 2:43 PM, Nathan Goldbaum <nathan12343@gmail.com> wrote:
On Sun, Jun 8, 2014 at 12:27 PM, Brendan Griffen <brendan.f.griffen@gmail.com> wrote:
Also, how do I construct just a zero filled yt array with dimensions (ndim,ndim,ndim)? Thanks
from yt import YTArray from numpy import np
arr = YTArray(np.zeros([ndim, ndim, ndim]), input_units=units_string)
or alternatively:
from yt.units import kiloparsec
arr = kiloparsec*np.zeros([ndim, ndim, ndim])
it doesn't have to be kiloparsec - you can compose the units you want out of any of the unit symbols that live in yt.units.
See this page for a ton more detail about yt's new unit system: http://yt-project.org/docs/dev-3.0/analyzing/units/index.html
Brendan
On Sun, Jun 8, 2014 at 3:26 PM, Brendan Griffen <brendan.f.griffen@gmail.com> wrote:
Hi,
Since I get memory errors. Could I not just read in the blocks of
wrote: the
output individually then basically stack the mesh each time. That way not every single particle of the snapshot has to be loaded at the same time. Would that just be a case of doing
level = int(math.log(ndim,2)) cg = ds.covering_grid(level=level, left_edge=[0,0,0],dims=[ndim,ndim,ndim]) arr = cg['deposit', 'all_density'] arrall += arr
in a loop over each HDF5 block?
It's likely that the memory use is dominated by the octree rather than the covering grid. This is with 1024^3 particles, correct?
You can probably significantly reduce the memory used by the octree by increasing n_ref in the call to load_particles.
See this page for more detail about load_particles:
http://yt-project.org/docs/dev-3.0/examining/loading_data.html#generic-parti...
Larger n_ref means fewer octree cells (lower resolution), but it also means lower poisson noise and lower memory use.
Alternatively, as Matt suggested, you could break your 1024^3 ensemble of particles up into chunks, loop over the chunk, creating a particle octree and then a covering grid for each subset of the particles. Your final covering grid is just the sub of the covering grids for each subset of particles.
Thanks. Brendan
On Fri, Jun 6, 2014 at 7:26 PM, Matthew Turk <matthewturk@gmail.com
wrote: > > > On Jun 6, 2014 4:54 PM, "Brendan Griffen" > <brendan.f.griffen@gmail.com> > wrote: > > > > OK great. It is very low resolution but it worked. Thanks for all > > your > > help. My higher resolution run 1024^3 in 100 Mpc seems to crash on > > 128GB > > memory machine. I might have to look elsewhere. > > > > If you are looking for low resolution extraction you can tune the > memory usage by changing the parameter n_ref to something higher. > > Supporting extremely large datasets in a single mesh is on the > roadmap > for the late summer or fall, after a 3.0 release goes out. For now > you can > also extract before you load in; this is sort of how we are > supporting an > INCITE project with very large particle counts. > > > > Also, I normally use Canopy distribution but I just use an alias to > > loadyt which erases my PYTHONPATH and I can't access scipy and a > > few other > > libraries any more. What is the best practice here? Should I just > > manually > > export PYTHONPATH and point to the libraries need in canopy or can > > they play > > nice together? > > > > Thanks. > > > > BG > > > > > > On Fri, Jun 6, 2014 at 2:54 PM, Nathan Goldbaum > > <nathan12343@gmail.com> wrote: > >> > >> > >> > >> > >> On Fri, Jun 6, 2014 at 11:48 AM, Brendan Griffen > >> <brendan.f.griffen@gmail.com> wrote: > >>> > >>> OK great. Thanks. I just wanted a homogeneous mesh. 512^3 with no > >>> nesting of any kind. Though when I plot the image it looks like > >>> it is > >>> assigning particles incorrectly (low resolution on the outside). > >>> This is > >>> just a test image. > >>> > >> > >> The SlicePlot is visualizing the octree so there is less > >> resolution > >> where there are fewer particles. If you want to visualize the > >> covering grid > >> you're going to need to visualize that separately. > >> > >>> > >>> ds = yt.load_particles(data, length_unit=3.08e24, > >>> mass_unit=1.9891e33,bbox=bbox) > >>> > >>> ad = ds.all_data() > >>> print ad['deposit', 'all_cic'] > >>> slc = yt.SlicePlot(ds, 2, ('deposit', 'all_cic')) > >>> slc.set_figure_size(4) > >>> cg = ds.covering_grid(level=9, > >>> left_edge=[0,0,0],dims=[512,512,512]) > >>> > >> > >> To actually produce the uniform resolution ndarray, you're going > >> to > >> need to do something like: > >> > >> array = cg[('deposit', 'all_cic')] > >> > >> array will then be a 3D array you can do whatever you want with. > >> By > >> default it has units, but to strip them off you'll just need to > >> cast to > >> ndarray: > >> > >> array_without_units = array.v > >> > >> > >>> > >>> Also, is there a way to load multiple particle types? > >>> > >>> Do I just need to stack the particles into the array here? > >>> > >>> data = {'particle_position_x': pos[:,0], > >>> 'particle_position_y': pos[:,1], > >>> 'particle_position_z': pos[:,2], > >>> 'particle_mass': np.array([mpart]*npart)} > >>> > >>> Then feed it in as usual? > >> > >> > >> That's right, although if the particle masses are different for > >> the > >> different particle types that code snippet will need to be > >> generalized to > >> handle that. > >> > >> I think in principle it should be possible to make load_particles > >> handle different particle types just like an SPH dataset that > >> contains > >> multiple particle types, but right now that hasn't been > >> implemented yet. > >> > >>> > >>> > >>> Brendan > >>> > >>> > >>> On Thu, Jun 5, 2014 at 9:44 PM, Nathan Goldbaum > >>> <nathan12343@gmail.com> wrote: > >>>> > >>>> That's right, you can set that via the bbox keyword parameter > >>>> for > >>>> load_particles. I'd urge you to take a look at the docstrings > >>>> and source > >>>> code for load_particles. > >>>> > >>>> > >>>> On Thu, Jun 5, 2014 at 6:34 PM, Brendan Griffen > >>>> <brendan.f.griffen@gmail.com> wrote: > >>>>> > >>>>> Thanks very much Nathan. I tried to load in my own data but I > >>>>> think there are too many particles or I have to specifically > >>>>> set the domain > >>>>> size. > >>>>> > >>>>> In this area: > >>>>> > >>>>> data = {'particle_position_x': pos[:,0], > >>>>> 'particle_position_y': pos[:,1], > >>>>> 'particle_position_z': pos[:,2], > >>>>> 'particle_mass': np.array([mpart]*npart)} > >>>>> > >>>>> ds = yt.load_particles(data, length_unit=3.08e24, > >>>>> mass_unit=1.9891e36) > >>>>> ad = ds.all_data() > >>>>> print ad['deposit', 'all_cic'] > >>>>> > >>>>> In [3]: run ytcic.py > >>>>> yt : [INFO ] 2014-06-05 21:29:06,183 Parameters: > >>>>> current_time > >>>>> = 0.0 > >>>>> yt : [INFO ] 2014-06-05 21:29:06,183 Parameters: > >>>>> domain_dimensions = [2 2 2] > >>>>> yt : [INFO ] 2014-06-05 21:29:06,184 Parameters: > >>>>> domain_left_edge = [ 0. 0. 0.] > >>>>> yt : [INFO ] 2014-06-05 21:29:06,185 Parameters: > >>>>> domain_right_edge = [ 1. 1. 1.] > >>>>> yt : [INFO ] 2014-06-05 21:29:06,185 Parameters: > >>>>> cosmological_simulation = 0.0 > >>>>> yt : [INFO ] 2014-06-05 21:29:06,188 Allocating for > >>>>> 1.342e+08 > >>>>> particles > >>>>> > >>>>> > >>>>>
> >>>>> YTDomainOverflow Traceback (most > >>>>> recent > >>>>> call last) > >>>>> > >>>>> > >>>>> /nfs/blank/h4231/bgriffen/data/lib/yt-x86_64/lib/python2.7/site-packages/IPython/utils/py3compat.pyc > >>>>> in execfile(fname, *where) > >>>>> 202 else: > >>>>> 203 filename = fname > >>>>> --> 204 __builtin__.execfile(filename, *where) > >>>>> > >>>>> > >>>>> > >>>>> /nfs/blank/h4231/bgriffen/work/projects/caterpillar/c2ray/cic/ytcic.py in > >>>>> <module>() > >>>>> 52 > >>>>> 53 ad = ds.all_data() > >>>>> ---> 54 print ad['deposit', 'all_cic'] > >>>>> 55 slc = yt.SlicePlot(ds, 2, ('deposit', 'all_cic')) > >>>>> 56 slc.set_figure_size(4) > >>>>> > >>>>> > >>>>> > >>>>> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc > >>>>> in __getitem__(self, key) > >>>>> 205 Returns a single field. Will add if necessary. > >>>>> 206 """ > >>>>> --> 207 f = self._determine_fields([key])[0] > >>>>> 208 if f not in self.field_data and key not in > >>>>> self.field_data: > >>>>> 209 if f in self._container_fields: > >>>>> > >>>>> > >>>>> > >>>>> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc > >>>>> in _determine_fields(self, fields) > >>>>> 453 raise YTFieldNotParseable(field) > >>>>> 454 ftype, fname = field > >>>>> --> 455 finfo = self.pf._get_field_info(ftype, > >>>>> fname) > >>>>> 456 else: > >>>>> 457 fname = field > >>>>> > >>>>> > >>>>> > >>>>> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/static_output.pyc > >>>>> in _get_field_info(self, ftype, fname) > >>>>> 445 _last_finfo = None > >>>>> 446 def _get_field_info(self, ftype, fname = None): > >>>>> --> 447 self.index > >>>>> 448 if fname is None: > >>>>> 449 ftype, fname = "unknown", ftype > >>>>> > >>>>> > >>>>> > >>>>> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/static_output.pyc > >>>>> in index(self) > >>>>> 277 raise RuntimeError("You should not > >>>>> instantiate Dataset.") > >>>>> 278 self._instantiated_index = > >>>>> self._index_class( > >>>>> --> 279 self, dataset_type=self.dataset_type) > >>>>> 280 # Now we do things that we need an > >>>>> instantiated index for > >>>>> 281 # ...first off, we create our field_info > >>>>> now. > >>>>> > >>>>> > >>>>> > >>>>> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/frontends/stream/data_structures.pyc > >>>>> in __init__(self, pf, dataset_type) > >>>>> 942 def __init__(self, pf, dataset_type = None): > >>>>> 943 self.stream_handler = pf.stream_handler > >>>>> --> 944 super(StreamParticleIndex, self).__init__(pf, > >>>>> dataset_type) > >>>>> 945 > >>>>> 946 def _setup_data_io(self): > >>>>> > >>>>> > >>>>> > >>>>> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/geometry/particle_geometry_handler.pyc > >>>>> in __init__(self, pf, dataset_type) > >>>>> 48 self.directory = > >>>>> os.path.dirname(self.index_filename) > >>>>> 49 self.float_type = np.float64 > >>>>> ---> 50 super(ParticleIndex, self).__init__(pf, > >>>>> dataset_type) > >>>>> 51 > >>>>> 52 def _setup_geometry(self): > >>>>> > >>>>> > >>>>> > >>>>> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/geometry/geometry_handler.pyc > >>>>> in __init__(self, pf, dataset_type) > >>>>> 54 > >>>>> 55 mylog.debug("Setting up domain geometry.") > >>>>> ---> 56 self._setup_geometry() > >>>>> 57 > >>>>> 58 mylog.debug("Initializing data grid data IO") > >>>>> > >>>>> > >>>>> > >>>>> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/geometry/particle_geometry_handler.pyc > >>>>> in _setup_geometry(self) > >>>>> 52 def _setup_geometry(self): > >>>>> 53 mylog.debug("Initializing Particle Geometry > >>>>> Handler.") > >>>>> ---> 54 self._initialize_particle_handler() > >>>>> 55 > >>>>> 56 > >>>>> > >>>>> > >>>>> > >>>>> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/geometry/particle_geometry_handler.pyc > >>>>> in _initialize_particle_handler(self) > >>>>> 87 pf.domain_left_edge, > >>>>> pf.domain_right_edge, > >>>>> 88 [N, N, N], len(self.data_files)) > >>>>> ---> 89 self._initialize_indices() > >>>>> 90 self.oct_handler.finalize() > >>>>> 91 self.max_level = self.oct_handler.max_level > >>>>> > >>>>> > >>>>> > >>>>> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/geometry/particle_geometry_handler.pyc > >>>>> in _initialize_indices(self) > >>>>> 109 npart = > >>>>> sum(data_file.total_particles.values()) > >>>>> 110 morton[ind:ind + npart] = \ > >>>>> --> 111 self.io._initialize_index(data_file, > >>>>> self.regions) > >>>>> 112 ind += npart > >>>>> 113 morton.sort() > >>>>> > >>>>> > >>>>> > >>>>> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/frontends/stream/io.pyc in > >>>>> _initialize_index(self, data_file, regions) > >>>>> 144 raise YTDomainOverflow(pos.min(axis=0), > >>>>> pos.max(axis=0), > >>>>> 145 > >>>>> data_file.pf.domain_left_edge, > >>>>> --> 146 > >>>>> data_file.pf.domain_right_edge) > >>>>> 147 regions.add_data_file(pos, > >>>>> data_file.file_id) > >>>>> 148 morton.append(compute_morton( > >>>>> > >>>>> YTDomainOverflow: Particle bounds [ 0. 0. 0.] and [ > >>>>> 99.99999237 > >>>>> 99.99999237 99.99999237] exceed domain bounds [ 0. 0. 0.] > >>>>> code_length and > >>>>> [ 1. 1. 1.] code_length > >>>>> > >>>>> > >>>>> > >>>>> On Thu, Jun 5, 2014 at 8:22 PM, Nathan Goldbaum > >>>>> <nathan12343@gmail.com> wrote: > >>>>>> > >>>>>> Here's a worked out example that does what you're looking for > >>>>>> using a fake 1 million particle dataset: > >>>>>> > >>>>>> > >>>>>> http://nbviewer.ipython.org/gist/ngoldbaum/546d37869aafe71cfe38 > >>>>>> > >>>>>> In this notebook I make use of two key yt features: > >>>>>> `load_particles`, and `covering_grid`. > >>>>>> > >>>>>> load_particles creates a "stream" dataset based on in-memory > >>>>>> data > >>>>>> fed in as a numpy array. This dataset acts just like an > >>>>>> on-disk simulation > >>>>>> dataset, but doesn't come with the baggage of needing to write > >>>>>> a custom > >>>>>> frontend to read a specific data format off disk. > >>>>>> > >>>>>> covering_grid is a way to generate uniform resolution data > >>>>>> from > >>>>>> an AMR dataset. It acts like a python dictionary where the > >>>>>> keys are field > >>>>>> names and returns 3D numpy arrays of whatever uniform > >>>>>> resolution you specify > >>>>>> when you create the covering_grid. > >>>>>> > >>>>>> Note that if you're using load_particles all of your data > >>>>>> needs > >>>>>> to live in memory. If your data is too big for that you'll > >>>>>> need to write a > >>>>>> frontend for your data format or use a memmap to an on-disk > >>>>>> file somehow. > >>>>>> I'm not an expert on that but others on the list should be > >>>>>> able to help out. > >>>>>> > >>>>>> Hope that gets you well on your way :) > >>>>>> > >>>>>> -Nathan > >>>>>> > >>>>>> > >>>>>> On Thu, Jun 5, 2014 at 5:04 PM, Desika Narayanan > >>>>>> <dnarayan@haverford.edu> wrote: > >>>>>>> > >>>>>>> Hey Brendan, > >>>>>>> > >>>>>>> A couple of extra tools you might find helpful in conjunction > >>>>>>> with Nathan's example of depositing the particles onto an > >>>>>>> octree are at: > >>>>>>> > >>>>>>> http://paste.yt-project.org/show/4737/ > >>>>>>> > >>>>>>> Where I load a gadget snapshot, and then recover the > >>>>>>> coordinates > >>>>>>> and width of each cell. > >>>>>>> > >>>>>>> In response to your last question - the particles are > >>>>>>> deposited > >>>>>>> into an octree grid (so, you'll see that the cell sizes > >>>>>>> aren't all the same > >>>>>>> size). I don't know if depositing onto a regular NxNxN mesh > >>>>>>> is possible, > >>>>>>> though would be interested to hear if so. > >>>>>>> > >>>>>>> -d > >>>>>>> > >>>>>>> > >>>>>>> On Thu, Jun 5, 2014 at 7:58 PM, Brendan Griffen > >>>>>>> <brendan.f.griffen@gmail.com> wrote: > >>>>>>>> > >>>>>>>> Thanks. I'll get the "bleeding edge" version first then try > >>>>>>>> your suggestions. Though I want to return the NxNxN array > >>>>>>>> and be able to > >>>>>>>> write this mesh to a file. It is *only* using the cic part > >>>>>>>> of yt and it > >>>>>>>> should return the mesh to be written? Just wanted to > >>>>>>>> clarify? > >>>>>>>> > >>>>>>>> Thanks. > >>>>>>>> Brendan > >>>>>>>> > >>>>>>>> > >>>>>>>> On Thu, Jun 5, 2014 at 6:49 PM, Nathan Goldbaum > >>>>>>>> <nathan12343@gmail.com> wrote: > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> On Thu, Jun 5, 2014 at 3:36 PM, John ZuHone > >>>>>>>>> <jzuhone@gmail.com> wrote: > >>>>>>>>>> > >>>>>>>>>> Hi Brendan, > >>>>>>>>>> > >>>>>>>>>> Which version of yt are you using? > >>>>>>>>>> > >>>>>>>>>> If you're using 3.0, this is actually fairly easy. If you > >>>>>>>>>> look in yt.fields.particle_fields.py, around line 85, you > >>>>>>>>>> can see how this > >>>>>>>>>> is done for the "particle_density" and "particle_mass" > >>>>>>>>>> fields. Basically you > >>>>>>>>>> can call a "deposit" method which takes the particle field > >>>>>>>>>> quantity you want > >>>>>>>>>> deposited and deposits it into cells. The underlying > >>>>>>>>>> calculation is done > >>>>>>>>>> using Cython, so it's fast. > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> And you shouldn't ever actually need to call these > >>>>>>>>> "deposit" > >>>>>>>>> functions, since "deposit" is exposed as a field type for > >>>>>>>>> all datasets that > >>>>>>>>> contain particles. > >>>>>>>>> > >>>>>>>>> Here is a notebook that does this for Enzo AMR data: > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> http://nbviewer.ipython.org/gist/ngoldbaum/5e19e4e6cc2bf330149c > >>>>>>>>> > >>>>>>>>> This dataset contains about a million particles and > >>>>>>>>> generates > >>>>>>>>> a CIC deposition for the whole domain in about 6 seconds > >>>>>>>>> from a cold start. > >>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> If you're using 2.x, then you can do the same thing, but > >>>>>>>>>> it's > >>>>>>>>>> not as straightforward. You can see how this works in > >>>>>>>>>> yt.data_objects.universal_fields.py, around line 986, > >>>>>>>>>> where the > >>>>>>>>>> "particle_density" field is defined. Basically, it calls > >>>>>>>>>> CICDeposit_3, which > >>>>>>>>>> is in yt.utilities.lib.CICDeposit.pyx. > >>>>>>>>>> > >>>>>>>>>> Let me know if you need any more clarification. > >>>>>>>>>> > >>>>>>>>>> Best, > >>>>>>>>>> > >>>>>>>>>> John Z > >>>>>>>>>> > >>>>>>>>>> On Jun 5, 2014, at 6:07 PM, Brendan Griffen > >>>>>>>>>> <brendan.f.griffen@gmail.com> wrote: > >>>>>>>>>> > >>>>>>>>>> > Hi, > >>>>>>>>>> > > >>>>>>>>>> > I was wondering if there were any Cython routines within > >>>>>>>>>> > yt > >>>>>>>>>> > which takes particle data and converts it into a > >>>>>>>>>> > cloud-in-cell based mesh > >>>>>>>>>> > which can be written to a file of my choosing. > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> What sort of mesh were you looking for? yt will internally > >>>>>>>>> construct an octree if it is fed particle data. I'm not > >>>>>>>>> sure whether this > >>>>>>>>> octree can be saved to disk for later analysis. > >>>>>>>>> > >>>>>>>>> It's also possible to create a uniform resolution covering > >>>>>>>>> grid containing field data for a deposited quantity, which > >>>>>>>>> can be quite > >>>>>>>>> easily saved to disk in a number of ways. > >>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> I heard a while ago there was some such functionality but > >>>>>>>>>> it > >>>>>>>>>> could be too far down the yt rabbit hole to be used as a > >>>>>>>>>> standalone? Is this > >>>>>>>>>> true? I have my own Python code for doing it but it just > >>>>>>>>>> isn't fast enough > >>>>>>>>>> and thought I'd ask the yt community if there were any > >>>>>>>>>> wrapper tools > >>>>>>>>>> available to boost the speed. > >>>>>>>>>> > > >>>>>>>>>> > Thanks. > >>>>>>>>>> > Brendan > >>>>>>>>>> > _______________________________________________ > >>>>>>>>>> > yt-users mailing list > >>>>>>>>>> > yt-users@lists.spacepope.org > >>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>> > http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org > >>>>>>>>>> > >>>>>>>>>> _______________________________________________ > >>>>>>>>>> yt-users mailing list > >>>>>>>>>> yt-users@lists.spacepope.org > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> _______________________________________________ > >>>>>>>>> yt-users mailing list > >>>>>>>>> yt-users@lists.spacepope.org > >>>>>>>>> > >>>>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org > >>>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> _______________________________________________ > >>>>>>>> yt-users mailing list > >>>>>>>> yt-users@lists.spacepope.org > >>>>>>>> > >>>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org > >>>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> _______________________________________________ > >>>>>>> yt-users mailing list > >>>>>>> yt-users@lists.spacepope.org > >>>>>>> > >>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org > >>>>>>> > >>>>>> > >>>>>> > >>>>>> _______________________________________________ > >>>>>> yt-users mailing list > >>>>>> yt-users@lists.spacepope.org > >>>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org > >>>>>> > >>>>> > >>>>> > >>>>> _______________________________________________ > >>>>> yt-users mailing list > >>>>> yt-users@lists.spacepope.org > >>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org > >>>>> > >>>> > >>>> > >>>> _______________________________________________ > >>>> yt-users mailing list > >>>> yt-users@lists.spacepope.org > >>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org > >>>> > >>> > >>> > >>> _______________________________________________ > >>> yt-users mailing list > >>> yt-users@lists.spacepope.org > >>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org > >>> > >> > >> > >> _______________________________________________ > >> yt-users mailing list > >> yt-users@lists.spacepope.org > >> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org > >> > > > > > > _______________________________________________ > > yt-users mailing list > > yt-users@lists.spacepope.org > > http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org > > > > > _______________________________________________ > yt-users mailing list > yt-users@lists.spacepope.org > http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
Hey Brendan, Could you try running your script using the memory_profiler module on pypi? Here's an example script that uses the memory profiler: http://paste.yt-project.org/show/4748/ and the output for that script: http://paste.yt-project.org/show/4749/ For what it's worth, it does indeed look like matt's suggestion to use load_uniform_grid is an option, and might be more memory efficient in the end since you will go directly to the uniform grid you want without creating an octree. Here's an example: http://paste.yt-project.org/show/4750/ Here's the memory usage information for that example: http://paste.yt-project.org/show/4751/ I used a 256^3 uniform grid with normally distributed random data - I'm not sure whether it will also be more memory efficient in your case. On Sun, Jun 8, 2014 at 9:32 PM, Brendan Griffen <brendan.f.griffen@gmail.com
wrote:
This is the full error if it helps at all? It is indeed, loading in all of the quantities.
Loading particles... --> Loading particle type: 1 yt : [INFO ] 2014-06-08 15:12:05,540 Parameters: current_time = 0.0 yt : [INFO ] 2014-06-08 15:12:05,540 Parameters: domain_dimensions = [2 2 2] yt : [INFO ] 2014-06-08 15:12:05,541 Parameters: domain_left_edge = [ 0. 0. 0.] yt : [INFO ] 2014-06-08 15:12:05,542 Parameters: domain_right_edge = [ 100. 100. 100.] yt : [INFO ] 2014-06-08 15:12:05,542 Parameters: cosmological_simulation = 0.0 yt : [INFO ] 2014-06-08 15:12:05,548 Allocating for 1.074e+09 particles yt : [INFO ] 2014-06-08 15:16:16,195 Identified 7.584e+07 octs yt : [INFO ] 2014-06-08 15:16:16,299 Loading field plugins. yt : [INFO ] 2014-06-08 15:16:16,299 Loaded angular_momentum (8 new fields) yt : [INFO ] 2014-06-08 15:16:16,299 Loaded astro (14 new fields) yt : [INFO ] 2014-06-08 15:16:16,300 Loaded cosmology (20 new fields) yt : [INFO ] 2014-06-08 15:16:16,300 Loaded fluid (56 new fields) yt : [INFO ] 2014-06-08 15:16:16,301 Loaded fluid_vector (88 new fields) yt : [INFO ] 2014-06-08 15:16:16,301 Loaded geometric (103 new fields) yt : [INFO ] 2014-06-08 15:16:16,301 Loaded local (103 new fields) yt : [INFO ] 2014-06-08 15:16:16,302 Loaded magnetic_field (109 new fields) yt : [INFO ] 2014-06-08 15:16:16,302 Loaded species (109 new fields) --------------------------------------------------------------------------- MemoryError Traceback (most recent call last) /nfs/blank/h4231/bgriffen/data/lib/yt-x86_64/lib/python2.7/site-packages/IPython/utils/py3compat.pyc in execfile(fname, *where) 202 else: 203 filename = fname --> 204 __builtin__.execfile(filename, *where)
/nfs/blank/h4231/bgriffen/work/projects/caterpillar/c2ray/cic/ytcic.py in <module>() 99 slc.set_figure_size(4) 100 slc.save() --> 101 102 for ndim in dimlist: 103 print
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc in __getitem__(self, key) 218 return self.field_data[f] 219 else: --> 220 self.get_data(f) 221 # fi.units is the unit expression string. We depend on the registry 222 # hanging off the dataset to define this unit object.
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc in get_data(self, fields) 627 628 fields_to_generate += gen_fluids + gen_particles --> 629 self._generate_fields(fields_to_generate) 630 631 def _generate_fields(self, fields_to_generate):
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc in _generate_fields(self, fields_to_generate) 644 fi = self.pf._get_field_info(*field) 645 try: --> 646 fd = self._generate_field(field) 647 if type(fd) == np.ndarray: 648 fd = self.pf.arr(fd, fi.units)
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc in _generate_field(self, field) 255 tr = self._generate_particle_field(field) 256 else: --> 257 tr = self._generate_fluid_field(field) 258 if tr is None: 259 raise YTCouldNotGenerateField(field, self.pf)
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc in _generate_fluid_field(self, field) 273 finfo.check_available(gen_obj) 274 except NeedsGridType as ngt_exception: --> 275 rv = self._generate_spatial_fluid(field, ngt_exception.ghost_zones) 276 else: 277 rv = finfo(gen_obj)
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc in _generate_spatial_fluid(self, field, ngz) 289 o = self._current_chunk.objs[0] 290 with o._activate_cache(): --> 291 ind += o.select(self.selector, self[field], rv, ind) 292 else: 293 chunks = self.index._chunk(self, "spatial", ngz = ngz)
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc in __getitem__(self, key) 218 return self.field_data[f] 219 else: --> 220 self.get_data(f) 221 # fi.units is the unit expression string. We depend on the registry 222 # hanging off the dataset to define this unit object.
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc in get_data(self, fields) 627 628 fields_to_generate += gen_fluids + gen_particles --> 629 self._generate_fields(fields_to_generate) 630 631 def _generate_fields(self, fields_to_generate):
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc in _generate_fields(self, fields_to_generate) 644 fi = self.pf._get_field_info(*field) 645 try: --> 646 fd = self._generate_field(field) 647 if type(fd) == np.ndarray: 648 fd = self.pf.arr(fd, fi.units)
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc in _generate_field(self, field) 255 tr = self._generate_particle_field(field) 256 else: --> 257 tr = self._generate_fluid_field(field) 258 if tr is None: 259 raise YTCouldNotGenerateField(field, self.pf)
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc in _generate_fluid_field(self, field) 275 rv = self._generate_spatial_fluid(field, ngt_exception.ghost_zones) 276 else: --> 277 rv = finfo(gen_obj) 278 return rv 279
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/fields/derived_field.pyc in __call__(self, data) 176 "for %s" % (self.name,)) 177 with self.unit_registry(data): --> 178 dd = self._function(self, data) 179 for field_name in data.keys(): 180 if field_name not in original_fields:
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/fields/particle_fields.pyc in particle_cic(field, data) 113 def particle_cic(field, data): 114 pos = data[ptype, coord_name] --> 115 d = data.deposit(pos, [data[ptype, mass_name]], method = "cic") 116 d = data.apply_units(d, data[ptype, mass_name].units) 117 d /= data["index", "cell_volume"]
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/octree_subset.pyc in deposit(self, positions, fields, method) 167 mylog.debug("Depositing %s (%s^3) particles into %s Octs", 168 positions.shape[0], positions.shape[0]**0.3333333, nvals[-1]) --> 169 pos = np.asarray(positions.convert_to_units("code_length"), 170 dtype="float64") 171 # We should not need the following if we know in advance all our fields
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/units/yt_array.pyc in convert_to_units(self, units) 366 367 self.units = new_units --> 368 self *= conversion_factor 369 return self 370
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/units/yt_array.pyc in __imul__(self, other) 667 """ See __mul__. """ 668 oth = sanitize_units_mul(self, other) --> 669 return np.multiply(self, oth, out=self) 670 671 def __div__(self, right_object):
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/units/yt_array.pyc in __array_wrap__(self, out_arr, context) 966 # casting to YTArray avoids creating a YTQuantity with size > 1 967 return YTArray(np.array(out_arr, unit)) --> 968 return ret_class(np.array(out_arr), unit) 969 970
MemoryError:
On Sun, Jun 8, 2014 at 10:50 PM, Matthew Turk <matthewturk@gmail.com> wrote:
Hi Brendan,
Hi Matt,
Thanks for your detailed email. Forgive my naivety but why do you need
On Sun, Jun 8, 2014 at 9:21 PM, Brendan Griffen <brendan.f.griffen@gmail.com> wrote: the
oct-tree in the first place? I have a my own fortran code for constructing a cloud in cell mesh and it uses very little overhead (just the n^3 grid and the particle data itself). I then calculate the dx,dy,dzs to the nearest 8 grid points and distribute accordingly in a omp loop which is done in a fraction of a second. Does the situation with yt come about (oct tree etc.) necessarily because of the way it handles particle data? Is it essentially used to map the particles to domains in the grid or something?
That's not naive at all. There are two reasons --
1) The octree is used for indexing for neighbor lookups and early-termination of region selection for particles 2) The octree is used to estimate the "required resolution" for any operation that requires a space-filling value. (i.e., any time that a particle becomes a volume.)
Projections in yt are adaptive, in that they project down to the finest appropriate resolution. There's also the "arbitrary_grid" operation, which does precisely what you're describing, but as it stands right now the octree gets constructed at time of instantiation of the indexing system. Thinking it over, you may be able to avoid that completely by not using load_particles and instead using load_uniform_grid and supplying your desired dimensions. The field names should be the same.
The machine I max memory on has 128GB and the snapshots are using 1024^3 particles. Do you have any idea of how much memory the oct-tree uses as
a
function of particle/grid number? I am going to try on a 256GB machine (though this is a bit of a hassle). I'll see how I go.
I am disappointed that it's blowing out your RAM. This week I will try to get some memory profiling done. Could you file a bug to this effect, which will help me track it? Peak memory usage during indexing should only be 64 bits * Nparticles, unless you're using load_particles, in which case all the fields will *also* have to be in memory. It's about 8 gigabytes per field. So, I think there's something going wrong.
Thanks.
Brendan
On Sun, Jun 8, 2014 at 6:25 PM, Matthew Turk <matthewturk@gmail.com>
Hi all,
I feel like I owe a brief explanation of why things are tricky right now, what we're planning on doing, and how we're experimenting and developing.
Presently, the particle geometry handlers build a single mesh from all particles in the dataset, along with a coarse bitmask that correlates files to regions in the domain. This requires the allocation of a single int64 array of size Nparticles, which is sorted in place and then fed into an octree construction algorithm that then spits back out the mesh. Each octree component contains 3 64-bit integers and eitehr a void pointer or a pointer to eight other octs. Increasing n_ref decreases the number of octs in this mesh; when smoothing operaitons are conducted, a second "index" mesh is created for looking up particles near mesh points. Mesh points are used for adaptive resolution smoothing and other "deposit particles on the grid somehow" operations (including SPH kernel).
Anyway, because right now it requires a global mesh to be constructed, this is expensive and requires holding a 64-bit integer in memory for each particle. I think if you're loading the particles in differently there is some additional overhead as well, but I'm still a bit surprised you OOM on a 1024^3 dataset.
In general, we don't *need* this global mesh; is can be constructed as required, which would speed up both the initial index phase as well as the final meshing process. I got about 50% of the way to implementing this last fall, but because of various concerns and deadlines I haven't finished it. I intend to get back to it probably in July, right after we put out a 3.0, so that we can have it in time for 3.1. In principle this will make the particle codes much more similar to ARTIO, in that the mesh will be constructed only as required and discarded when no longer required, which will make them much more memory efficient.
But, getting a single mesh for extremely large data is a very high priority; right now for the 10240^3 run we've been loading up individual sub-chunks, which I want to stop doing.
From the technical perspective, these are the things that need to happen on the yt side for particle datasets to move to this "lazy" mode of loading; most of this is based on things learned from 2HOT and ARTIO, and will involve converting to a forest-of-octrees.
* Split into spatially-organized subchunks of ParticleOctreeSubset objects, such that these map 1:Nfiles, and that can be constructed on the fly. * Construct a dual-mesh of the bitmask "ParticleRegion" object that will help with identifying neighbors to a given oct cell, so that if we're inside one octree we know which neighbor octrees to grab if we need particles for smoothing things (fast boundary particle identification is later down the road) * Parallel sort of particles, or using the parallel ring function; may not be necessary after all
All of this is doable, and I'd be happy to work with people if they'd like to take a shot at implementing it, but I've mostly put it on my list for post-3.0.
-Matt
On Sun, Jun 8, 2014 at 2:43 PM, Nathan Goldbaum <nathan12343@gmail.com
wrote:
On Sun, Jun 8, 2014 at 12:27 PM, Brendan Griffen <brendan.f.griffen@gmail.com> wrote:
Also, how do I construct just a zero filled yt array with dimensions (ndim,ndim,ndim)? Thanks
from yt import YTArray from numpy import np
arr = YTArray(np.zeros([ndim, ndim, ndim]), input_units=units_string)
or alternatively:
from yt.units import kiloparsec
arr = kiloparsec*np.zeros([ndim, ndim, ndim])
it doesn't have to be kiloparsec - you can compose the units you want out of any of the unit symbols that live in yt.units.
See this page for a ton more detail about yt's new unit system: http://yt-project.org/docs/dev-3.0/analyzing/units/index.html
Brendan
On Sun, Jun 8, 2014 at 3:26 PM, Brendan Griffen <brendan.f.griffen@gmail.com> wrote: > > Hi, > > Since I get memory errors. Could I not just read in the blocks of
> output individually then basically stack the mesh each time. That way > not > every single particle of the snapshot has to be loaded at the same > time. > Would that just be a case of doing > > level = int(math.log(ndim,2)) > cg = ds.covering_grid(level=level, > left_edge=[0,0,0],dims=[ndim,ndim,ndim]) > arr = cg['deposit', 'all_density'] > arrall += arr > > in a loop over each HDF5 block?
It's likely that the memory use is dominated by the octree rather
the covering grid. This is with 1024^3 particles, correct?
You can probably significantly reduce the memory used by the octree by increasing n_ref in the call to load_particles.
See this page for more detail about load_particles:
http://yt-project.org/docs/dev-3.0/examining/loading_data.html#generic-parti...
Larger n_ref means fewer octree cells (lower resolution), but it also means lower poisson noise and lower memory use.
Alternatively, as Matt suggested, you could break your 1024^3
ensemble
of particles up into chunks, loop over the chunk, creating a particle octree and then a covering grid for each subset of the particles. Your final covering grid is just the sub of the covering grids for each subset of particles.
> > > Thanks. > Brendan > > > > > On Fri, Jun 6, 2014 at 7:26 PM, Matthew Turk < matthewturk@gmail.com> > wrote: >> >> >> On Jun 6, 2014 4:54 PM, "Brendan Griffen" >> <brendan.f.griffen@gmail.com> >> wrote: >> > >> > OK great. It is very low resolution but it worked. Thanks for all >> > your >> > help. My higher resolution run 1024^3 in 100 Mpc seems to crash on >> > 128GB >> > memory machine. I might have to look elsewhere. >> > >> >> If you are looking for low resolution extraction you can tune the >> memory usage by changing the parameter n_ref to something higher. >> >> Supporting extremely large datasets in a single mesh is on the >> roadmap >> for the late summer or fall, after a 3.0 release goes out. For now >> you can >> also extract before you load in; this is sort of how we are >> supporting an >> INCITE project with very large particle counts. >> >> >> > Also, I normally use Canopy distribution but I just use an alias to >> > loadyt which erases my PYTHONPATH and I can't access scipy and a >> > few other >> > libraries any more. What is the best practice here? Should I just >> > manually >> > export PYTHONPATH and point to the libraries need in canopy or can >> > they play >> > nice together? >> > >> > Thanks. >> > >> > BG >> > >> > >> > On Fri, Jun 6, 2014 at 2:54 PM, Nathan Goldbaum >> > <nathan12343@gmail.com> wrote: >> >> >> >> >> >> >> >> >> >> On Fri, Jun 6, 2014 at 11:48 AM, Brendan Griffen >> >> <brendan.f.griffen@gmail.com> wrote: >> >>> >> >>> OK great. Thanks. I just wanted a homogeneous mesh. 512^3 with no >> >>> nesting of any kind. Though when I plot the image it looks
wrote: the than like
>> >>> it is >> >>> assigning particles incorrectly (low resolution on the outside). >> >>> This is >> >>> just a test image. >> >>> >> >> >> >> The SlicePlot is visualizing the octree so there is less >> >> resolution >> >> where there are fewer particles. If you want to visualize the >> >> covering grid >> >> you're going to need to visualize that separately. >> >> >> >>> >> >>> ds = yt.load_particles(data, length_unit=3.08e24, >> >>> mass_unit=1.9891e33,bbox=bbox) >> >>> >> >>> ad = ds.all_data() >> >>> print ad['deposit', 'all_cic'] >> >>> slc = yt.SlicePlot(ds, 2, ('deposit', 'all_cic')) >> >>> slc.set_figure_size(4) >> >>> cg = ds.covering_grid(level=9, >> >>> left_edge=[0,0,0],dims=[512,512,512]) >> >>> >> >> >> >> To actually produce the uniform resolution ndarray, you're going >> >> to >> >> need to do something like: >> >> >> >> array = cg[('deposit', 'all_cic')] >> >> >> >> array will then be a 3D array you can do whatever you want with. >> >> By >> >> default it has units, but to strip them off you'll just need to >> >> cast to >> >> ndarray: >> >> >> >> array_without_units = array.v >> >> >> >> >> >>> >> >>> Also, is there a way to load multiple particle types? >> >>> >> >>> Do I just need to stack the particles into the array here? >> >>> >> >>> data = {'particle_position_x': pos[:,0], >> >>> 'particle_position_y': pos[:,1], >> >>> 'particle_position_z': pos[:,2], >> >>> 'particle_mass': np.array([mpart]*npart)} >> >>> >> >>> Then feed it in as usual? >> >> >> >> >> >> That's right, although if the particle masses are different for >> >> the >> >> different particle types that code snippet will need to be >> >> generalized to >> >> handle that. >> >> >> >> I think in principle it should be possible to make load_particles >> >> handle different particle types just like an SPH dataset that >> >> contains >> >> multiple particle types, but right now that hasn't been >> >> implemented yet. >> >> >> >>> >> >>> >> >>> Brendan >> >>> >> >>> >> >>> On Thu, Jun 5, 2014 at 9:44 PM, Nathan Goldbaum >> >>> <nathan12343@gmail.com> wrote: >> >>>> >> >>>> That's right, you can set that via the bbox keyword parameter >> >>>> for >> >>>> load_particles. I'd urge you to take a look at the docstrings >> >>>> and source >> >>>> code for load_particles. >> >>>> >> >>>> >> >>>> On Thu, Jun 5, 2014 at 6:34 PM, Brendan Griffen >> >>>> <brendan.f.griffen@gmail.com> wrote: >> >>>>> >> >>>>> Thanks very much Nathan. I tried to load in my own data but I >> >>>>> think there are too many particles or I have to specifically >> >>>>> set the domain >> >>>>> size. >> >>>>> >> >>>>> In this area: >> >>>>> >> >>>>> data = {'particle_position_x': pos[:,0], >> >>>>> 'particle_position_y': pos[:,1], >> >>>>> 'particle_position_z': pos[:,2], >> >>>>> 'particle_mass': np.array([mpart]*npart)} >> >>>>> >> >>>>> ds = yt.load_particles(data, length_unit=3.08e24, >> >>>>> mass_unit=1.9891e36) >> >>>>> ad = ds.all_data() >> >>>>> print ad['deposit', 'all_cic'] >> >>>>> >> >>>>> In [3]: run ytcic.py >> >>>>> yt : [INFO ] 2014-06-05 21:29:06,183 Parameters: >> >>>>> current_time >> >>>>> = 0.0 >> >>>>> yt : [INFO ] 2014-06-05 21:29:06,183 Parameters: >> >>>>> domain_dimensions = [2 2 2] >> >>>>> yt : [INFO ] 2014-06-05 21:29:06,184 Parameters: >> >>>>> domain_left_edge = [ 0. 0. 0.] >> >>>>> yt : [INFO ] 2014-06-05 21:29:06,185 Parameters: >> >>>>> domain_right_edge = [ 1. 1. 1.] >> >>>>> yt : [INFO ] 2014-06-05 21:29:06,185 Parameters: >> >>>>> cosmological_simulation = 0.0 >> >>>>> yt : [INFO ] 2014-06-05 21:29:06,188 Allocating for >> >>>>> 1.342e+08 >> >>>>> particles >> >>>>> >> >>>>> >> >>>>>
>> >>>>> YTDomainOverflow Traceback (most >> >>>>> recent >> >>>>> call last) >> >>>>> >> >>>>> >> >>>>> /nfs/blank/h4231/bgriffen/data/lib/yt-x86_64/lib/python2.7/site-packages/IPython/utils/py3compat.pyc >> >>>>> in execfile(fname, *where) >> >>>>> 202 else: >> >>>>> 203 filename = fname >> >>>>> --> 204 __builtin__.execfile(filename, *where) >> >>>>> >> >>>>> >> >>>>> >> >>>>> /nfs/blank/h4231/bgriffen/work/projects/caterpillar/c2ray/cic/ytcic.py in >> >>>>> <module>() >> >>>>> 52 >> >>>>> 53 ad = ds.all_data() >> >>>>> ---> 54 print ad['deposit', 'all_cic'] >> >>>>> 55 slc = yt.SlicePlot(ds, 2, ('deposit', 'all_cic')) >> >>>>> 56 slc.set_figure_size(4) >> >>>>> >> >>>>> >> >>>>> >> >>>>> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc >> >>>>> in __getitem__(self, key) >> >>>>> 205 Returns a single field. Will add if necessary. >> >>>>> 206 """ >> >>>>> --> 207 f = self._determine_fields([key])[0] >> >>>>> 208 if f not in self.field_data and key not in >> >>>>> self.field_data: >> >>>>> 209 if f in self._container_fields: >> >>>>> >> >>>>> >> >>>>> >> >>>>> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc >> >>>>> in _determine_fields(self, fields) >> >>>>> 453 raise YTFieldNotParseable(field) >> >>>>> 454 ftype, fname = field >> >>>>> --> 455 finfo = self.pf._get_field_info(ftype, >> >>>>> fname) >> >>>>> 456 else: >> >>>>> 457 fname = field >> >>>>> >> >>>>> >> >>>>> >> >>>>> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/static_output.pyc >> >>>>> in _get_field_info(self, ftype, fname) >> >>>>> 445 _last_finfo = None >> >>>>> 446 def _get_field_info(self, ftype, fname = None): >> >>>>> --> 447 self.index >> >>>>> 448 if fname is None: >> >>>>> 449 ftype, fname = "unknown", ftype >> >>>>> >> >>>>> >> >>>>> >> >>>>> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/static_output.pyc >> >>>>> in index(self) >> >>>>> 277 raise RuntimeError("You should not >> >>>>> instantiate Dataset.") >> >>>>> 278 self._instantiated_index = >> >>>>> self._index_class( >> >>>>> --> 279 self, dataset_type=self.dataset_type) >> >>>>> 280 # Now we do things that we need an >> >>>>> instantiated index for >> >>>>> 281 # ...first off, we create our field_info >> >>>>> now. >> >>>>> >> >>>>> >> >>>>> >> >>>>> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/frontends/stream/data_structures.pyc >> >>>>> in __init__(self, pf, dataset_type) >> >>>>> 942 def __init__(self, pf, dataset_type = None): >> >>>>> 943 self.stream_handler = pf.stream_handler >> >>>>> --> 944 super(StreamParticleIndex, self).__init__(pf, >> >>>>> dataset_type) >> >>>>> 945 >> >>>>> 946 def _setup_data_io(self): >> >>>>> >> >>>>> >> >>>>> >> >>>>> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/geometry/particle_geometry_handler.pyc >> >>>>> in __init__(self, pf, dataset_type) >> >>>>> 48 self.directory = >> >>>>> os.path.dirname(self.index_filename) >> >>>>> 49 self.float_type = np.float64 >> >>>>> ---> 50 super(ParticleIndex, self).__init__(pf, >> >>>>> dataset_type) >> >>>>> 51 >> >>>>> 52 def _setup_geometry(self): >> >>>>> >> >>>>> >> >>>>> >> >>>>> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/geometry/geometry_handler.pyc >> >>>>> in __init__(self, pf, dataset_type) >> >>>>> 54 >> >>>>> 55 mylog.debug("Setting up domain geometry.") >> >>>>> ---> 56 self._setup_geometry() >> >>>>> 57 >> >>>>> 58 mylog.debug("Initializing data grid data IO") >> >>>>> >> >>>>> >> >>>>> >> >>>>> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/geometry/particle_geometry_handler.pyc >> >>>>> in _setup_geometry(self) >> >>>>> 52 def _setup_geometry(self): >> >>>>> 53 mylog.debug("Initializing Particle Geometry >> >>>>> Handler.") >> >>>>> ---> 54 self._initialize_particle_handler() >> >>>>> 55 >> >>>>> 56 >> >>>>> >> >>>>> >> >>>>> >> >>>>> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/geometry/particle_geometry_handler.pyc >> >>>>> in _initialize_particle_handler(self) >> >>>>> 87 pf.domain_left_edge, >> >>>>> pf.domain_right_edge, >> >>>>> 88 [N, N, N], len(self.data_files)) >> >>>>> ---> 89 self._initialize_indices() >> >>>>> 90 self.oct_handler.finalize() >> >>>>> 91 self.max_level = self.oct_handler.max_level >> >>>>> >> >>>>> >> >>>>> >> >>>>> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/geometry/particle_geometry_handler.pyc >> >>>>> in _initialize_indices(self) >> >>>>> 109 npart = >> >>>>> sum(data_file.total_particles.values()) >> >>>>> 110 morton[ind:ind + npart] = \ >> >>>>> --> 111 self.io._initialize_index(data_file, >> >>>>> self.regions) >> >>>>> 112 ind += npart >> >>>>> 113 morton.sort() >> >>>>> >> >>>>> >> >>>>> >> >>>>> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/frontends/stream/io.pyc in >> >>>>> _initialize_index(self, data_file, regions) >> >>>>> 144 raise YTDomainOverflow(pos.min(axis=0), >> >>>>> pos.max(axis=0), >> >>>>> 145 >> >>>>> data_file.pf.domain_left_edge, >> >>>>> --> 146 >> >>>>> data_file.pf.domain_right_edge) >> >>>>> 147 regions.add_data_file(pos, >> >>>>> data_file.file_id) >> >>>>> 148 morton.append(compute_morton( >> >>>>> >> >>>>> YTDomainOverflow: Particle bounds [ 0. 0. 0.] and [ >> >>>>> 99.99999237 >> >>>>> 99.99999237 99.99999237] exceed domain bounds [ 0. 0. 0.] >> >>>>> code_length and >> >>>>> [ 1. 1. 1.] code_length >> >>>>> >> >>>>> >> >>>>> >> >>>>> On Thu, Jun 5, 2014 at 8:22 PM, Nathan Goldbaum >> >>>>> <nathan12343@gmail.com> wrote: >> >>>>>> >> >>>>>> Here's a worked out example that does what you're looking for >> >>>>>> using a fake 1 million particle dataset: >> >>>>>> >> >>>>>> >> >>>>>> http://nbviewer.ipython.org/gist/ngoldbaum/546d37869aafe71cfe38 >> >>>>>> >> >>>>>> In this notebook I make use of two key yt features: >> >>>>>> `load_particles`, and `covering_grid`. >> >>>>>> >> >>>>>> load_particles creates a "stream" dataset based on in-memory >> >>>>>> data >> >>>>>> fed in as a numpy array. This dataset acts just like an >> >>>>>> on-disk simulation >> >>>>>> dataset, but doesn't come with the baggage of needing to write >> >>>>>> a custom >> >>>>>> frontend to read a specific data format off disk. >> >>>>>> >> >>>>>> covering_grid is a way to generate uniform resolution data >> >>>>>> from >> >>>>>> an AMR dataset. It acts like a python dictionary where the >> >>>>>> keys are field >> >>>>>> names and returns 3D numpy arrays of whatever uniform >> >>>>>> resolution you specify >> >>>>>> when you create the covering_grid. >> >>>>>> >> >>>>>> Note that if you're using load_particles all of your data >> >>>>>> needs >> >>>>>> to live in memory. If your data is too big for that you'll >> >>>>>> need to write a >> >>>>>> frontend for your data format or use a memmap to an on-disk >> >>>>>> file somehow. >> >>>>>> I'm not an expert on that but others on the list should be >> >>>>>> able to help out. >> >>>>>> >> >>>>>> Hope that gets you well on your way :) >> >>>>>> >> >>>>>> -Nathan >> >>>>>> >> >>>>>> >> >>>>>> On Thu, Jun 5, 2014 at 5:04 PM, Desika Narayanan >> >>>>>> <dnarayan@haverford.edu> wrote: >> >>>>>>> >> >>>>>>> Hey Brendan, >> >>>>>>> >> >>>>>>> A couple of extra tools you might find helpful in conjunction >> >>>>>>> with Nathan's example of depositing the particles onto an >> >>>>>>> octree are at: >> >>>>>>> >> >>>>>>> http://paste.yt-project.org/show/4737/ >> >>>>>>> >> >>>>>>> Where I load a gadget snapshot, and then recover the >> >>>>>>> coordinates >> >>>>>>> and width of each cell. >> >>>>>>> >> >>>>>>> In response to your last question - the particles are >> >>>>>>> deposited >> >>>>>>> into an octree grid (so, you'll see that the cell sizes >> >>>>>>> aren't all the same >> >>>>>>> size). I don't know if depositing onto a regular NxNxN mesh >> >>>>>>> is possible, >> >>>>>>> though would be interested to hear if so. >> >>>>>>> >> >>>>>>> -d >> >>>>>>> >> >>>>>>> >> >>>>>>> On Thu, Jun 5, 2014 at 7:58 PM, Brendan Griffen >> >>>>>>> <brendan.f.griffen@gmail.com> wrote: >> >>>>>>>> >> >>>>>>>> Thanks. I'll get the "bleeding edge" version first then try >> >>>>>>>> your suggestions. Though I want to return the NxNxN array >> >>>>>>>> and be able to >> >>>>>>>> write this mesh to a file. It is *only* using the cic part >> >>>>>>>> of yt and it >> >>>>>>>> should return the mesh to be written? Just wanted to >> >>>>>>>> clarify? >> >>>>>>>> >> >>>>>>>> Thanks. >> >>>>>>>> Brendan >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> On Thu, Jun 5, 2014 at 6:49 PM, Nathan Goldbaum >> >>>>>>>> <nathan12343@gmail.com> wrote: >> >>>>>>>>> >> >>>>>>>>> >> >>>>>>>>> >> >>>>>>>>> >> >>>>>>>>> On Thu, Jun 5, 2014 at 3:36 PM, John ZuHone >> >>>>>>>>> <jzuhone@gmail.com> wrote: >> >>>>>>>>>> >> >>>>>>>>>> Hi Brendan, >> >>>>>>>>>> >> >>>>>>>>>> Which version of yt are you using? >> >>>>>>>>>> >> >>>>>>>>>> If you're using 3.0, this is actually fairly easy. If you >> >>>>>>>>>> look in yt.fields.particle_fields.py, around line 85, you >> >>>>>>>>>> can see how this >> >>>>>>>>>> is done for the "particle_density" and "particle_mass" >> >>>>>>>>>> fields. Basically you >> >>>>>>>>>> can call a "deposit" method which takes the particle field >> >>>>>>>>>> quantity you want >> >>>>>>>>>> deposited and deposits it into cells. The underlying >> >>>>>>>>>> calculation is done >> >>>>>>>>>> using Cython, so it's fast. >> >>>>>>>>> >> >>>>>>>>> >> >>>>>>>>> And you shouldn't ever actually need to call these >> >>>>>>>>> "deposit" >> >>>>>>>>> functions, since "deposit" is exposed as a field type for >> >>>>>>>>> all datasets that >> >>>>>>>>> contain particles. >> >>>>>>>>> >> >>>>>>>>> Here is a notebook that does this for Enzo AMR data: >> >>>>>>>>> >> >>>>>>>>> >> >>>>>>>>> >> >>>>>>>>> http://nbviewer.ipython.org/gist/ngoldbaum/5e19e4e6cc2bf330149c >> >>>>>>>>> >> >>>>>>>>> This dataset contains about a million particles and >> >>>>>>>>> generates >> >>>>>>>>> a CIC deposition for the whole domain in about 6 seconds >> >>>>>>>>> from a cold start. >> >>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>>> If you're using 2.x, then you can do the same thing, but >> >>>>>>>>>> it's >> >>>>>>>>>> not as straightforward. You can see how this works in >> >>>>>>>>>> yt.data_objects.universal_fields.py, around line 986, >> >>>>>>>>>> where the >> >>>>>>>>>> "particle_density" field is defined. Basically, it calls >> >>>>>>>>>> CICDeposit_3, which >> >>>>>>>>>> is in yt.utilities.lib.CICDeposit.pyx. >> >>>>>>>>>> >> >>>>>>>>>> Let me know if you need any more clarification. >> >>>>>>>>>> >> >>>>>>>>>> Best, >> >>>>>>>>>> >> >>>>>>>>>> John Z >> >>>>>>>>>> >> >>>>>>>>>> On Jun 5, 2014, at 6:07 PM, Brendan Griffen >> >>>>>>>>>> <brendan.f.griffen@gmail.com> wrote: >> >>>>>>>>>> >> >>>>>>>>>> > Hi, >> >>>>>>>>>> > >> >>>>>>>>>> > I was wondering if there were any Cython routines within >> >>>>>>>>>> > yt >> >>>>>>>>>> > which takes particle data and converts it into a >> >>>>>>>>>> > cloud-in-cell based mesh >> >>>>>>>>>> > which can be written to a file of my choosing. >> >>>>>>>>> >> >>>>>>>>> >> >>>>>>>>> What sort of mesh were you looking for? yt will internally >> >>>>>>>>> construct an octree if it is fed particle data. I'm not >> >>>>>>>>> sure whether this >> >>>>>>>>> octree can be saved to disk for later analysis. >> >>>>>>>>> >> >>>>>>>>> It's also possible to create a uniform resolution covering >> >>>>>>>>> grid containing field data for a deposited quantity, which >> >>>>>>>>> can be quite >> >>>>>>>>> easily saved to disk in a number of ways. >> >>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>>> I heard a while ago there was some such functionality but >> >>>>>>>>>> it >> >>>>>>>>>> could be too far down the yt rabbit hole to be used as a >> >>>>>>>>>> standalone? Is this >> >>>>>>>>>> true? I have my own Python code for doing it but it just >> >>>>>>>>>> isn't fast enough >> >>>>>>>>>> and thought I'd ask the yt community if there were any >> >>>>>>>>>> wrapper tools >> >>>>>>>>>> available to boost the speed. >> >>>>>>>>>> > >> >>>>>>>>>> > Thanks. >> >>>>>>>>>> > Brendan >> >>>>>>>>>> > _______________________________________________ >> >>>>>>>>>> > yt-users mailing list >> >>>>>>>>>> > yt-users@lists.spacepope.org >> >>>>>>>>>> > >> >>>>>>>>>> > >> >>>>>>>>>> > http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >> >>>>>>>>>> >> >>>>>>>>>> _______________________________________________ >> >>>>>>>>>> yt-users mailing list >> >>>>>>>>>> yt-users@lists.spacepope.org >> >>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >> >>>>>>>>> >> >>>>>>>>> >> >>>>>>>>> >> >>>>>>>>> _______________________________________________ >> >>>>>>>>> yt-users mailing list >> >>>>>>>>> yt-users@lists.spacepope.org >> >>>>>>>>> >> >>>>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >> >>>>>>>>> >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> _______________________________________________ >> >>>>>>>> yt-users mailing list >> >>>>>>>> yt-users@lists.spacepope.org >> >>>>>>>> >> >>>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >> >>>>>>>> >> >>>>>>> >> >>>>>>> >> >>>>>>> _______________________________________________ >> >>>>>>> yt-users mailing list >> >>>>>>> yt-users@lists.spacepope.org >> >>>>>>> >> >>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >> >>>>>>> >> >>>>>> >> >>>>>> >> >>>>>> _______________________________________________ >> >>>>>> yt-users mailing list >> >>>>>> yt-users@lists.spacepope.org >> >>>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >> >>>>>> >> >>>>> >> >>>>> >> >>>>> _______________________________________________ >> >>>>> yt-users mailing list >> >>>>> yt-users@lists.spacepope.org >> >>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >> >>>>> >> >>>> >> >>>> >> >>>> _______________________________________________ >> >>>> yt-users mailing list >> >>>> yt-users@lists.spacepope.org >> >>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >> >>>> >> >>> >> >>> >> >>> _______________________________________________ >> >>> yt-users mailing list >> >>> yt-users@lists.spacepope.org >> >>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >> >>> >> >> >> >> >> >> _______________________________________________ >> >> yt-users mailing list >> >> yt-users@lists.spacepope.org >> >> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >> >> >> > >> > >> > _______________________________________________ >> > yt-users mailing list >> > yt-users@lists.spacepope.org >> > http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >> > >> >> >> _______________________________________________ >> yt-users mailing list >> yt-users@lists.spacepope.org >> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >> >
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
Thanks Nathan. Comparing the two though: 26 1536.664 MiB 1130.480 MiB uniform_array = grid_object['deposit', 'all_cic'] 24 1285.703 MiB 890.539 MiB cic_density = ad["deposit", "all_cic"] Isn't it the uniform grid which uses the most memory? I'm running the profiling now and will get back to you. I hope in spite of the fact that it does crash it will still give me some useful output. Brendan On Mon, Jun 9, 2014 at 2:05 AM, Nathan Goldbaum <nathan12343@gmail.com> wrote:
Hey Brendan,
Could you try running your script using the memory_profiler module on pypi?
Here's an example script that uses the memory profiler: http://paste.yt-project.org/show/4748/
and the output for that script: http://paste.yt-project.org/show/4749/
For what it's worth, it does indeed look like matt's suggestion to use load_uniform_grid is an option, and might be more memory efficient in the end since you will go directly to the uniform grid you want without creating an octree. Here's an example: http://paste.yt-project.org/show/4750/
Here's the memory usage information for that example: http://paste.yt-project.org/show/4751/
I used a 256^3 uniform grid with normally distributed random data - I'm not sure whether it will also be more memory efficient in your case.
On Sun, Jun 8, 2014 at 9:32 PM, Brendan Griffen < brendan.f.griffen@gmail.com> wrote:
This is the full error if it helps at all? It is indeed, loading in all of the quantities.
Loading particles... --> Loading particle type: 1 yt : [INFO ] 2014-06-08 15:12:05,540 Parameters: current_time = 0.0 yt : [INFO ] 2014-06-08 15:12:05,540 Parameters: domain_dimensions = [2 2 2] yt : [INFO ] 2014-06-08 15:12:05,541 Parameters: domain_left_edge = [ 0. 0. 0.] yt : [INFO ] 2014-06-08 15:12:05,542 Parameters: domain_right_edge = [ 100. 100. 100.] yt : [INFO ] 2014-06-08 15:12:05,542 Parameters: cosmological_simulation = 0.0 yt : [INFO ] 2014-06-08 15:12:05,548 Allocating for 1.074e+09 particles yt : [INFO ] 2014-06-08 15:16:16,195 Identified 7.584e+07 octs yt : [INFO ] 2014-06-08 15:16:16,299 Loading field plugins. yt : [INFO ] 2014-06-08 15:16:16,299 Loaded angular_momentum (8 new fields) yt : [INFO ] 2014-06-08 15:16:16,299 Loaded astro (14 new fields) yt : [INFO ] 2014-06-08 15:16:16,300 Loaded cosmology (20 new fields) yt : [INFO ] 2014-06-08 15:16:16,300 Loaded fluid (56 new fields) yt : [INFO ] 2014-06-08 15:16:16,301 Loaded fluid_vector (88 new fields) yt : [INFO ] 2014-06-08 15:16:16,301 Loaded geometric (103 new fields) yt : [INFO ] 2014-06-08 15:16:16,301 Loaded local (103 new fields) yt : [INFO ] 2014-06-08 15:16:16,302 Loaded magnetic_field (109 new fields) yt : [INFO ] 2014-06-08 15:16:16,302 Loaded species (109 new fields)
--------------------------------------------------------------------------- MemoryError Traceback (most recent call last) /nfs/blank/h4231/bgriffen/data/lib/yt-x86_64/lib/python2.7/site-packages/IPython/utils/py3compat.pyc in execfile(fname, *where) 202 else: 203 filename = fname --> 204 __builtin__.execfile(filename, *where)
/nfs/blank/h4231/bgriffen/work/projects/caterpillar/c2ray/cic/ytcic.py in <module>() 99 slc.set_figure_size(4) 100 slc.save() --> 101 102 for ndim in dimlist: 103 print
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc in __getitem__(self, key) 218 return self.field_data[f] 219 else: --> 220 self.get_data(f) 221 # fi.units is the unit expression string. We depend on the registry 222 # hanging off the dataset to define this unit object.
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc in get_data(self, fields) 627 628 fields_to_generate += gen_fluids + gen_particles --> 629 self._generate_fields(fields_to_generate) 630 631 def _generate_fields(self, fields_to_generate):
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc in _generate_fields(self, fields_to_generate) 644 fi = self.pf._get_field_info(*field) 645 try: --> 646 fd = self._generate_field(field) 647 if type(fd) == np.ndarray: 648 fd = self.pf.arr(fd, fi.units)
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc in _generate_field(self, field) 255 tr = self._generate_particle_field(field) 256 else: --> 257 tr = self._generate_fluid_field(field) 258 if tr is None: 259 raise YTCouldNotGenerateField(field, self.pf)
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc in _generate_fluid_field(self, field) 273 finfo.check_available(gen_obj) 274 except NeedsGridType as ngt_exception: --> 275 rv = self._generate_spatial_fluid(field, ngt_exception.ghost_zones) 276 else: 277 rv = finfo(gen_obj)
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc in _generate_spatial_fluid(self, field, ngz) 289 o = self._current_chunk.objs[0] 290 with o._activate_cache(): --> 291 ind += o.select(self.selector, self[field], rv, ind) 292 else: 293 chunks = self.index._chunk(self, "spatial", ngz = ngz)
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc in __getitem__(self, key) 218 return self.field_data[f] 219 else: --> 220 self.get_data(f) 221 # fi.units is the unit expression string. We depend on the registry 222 # hanging off the dataset to define this unit object.
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc in get_data(self, fields) 627 628 fields_to_generate += gen_fluids + gen_particles --> 629 self._generate_fields(fields_to_generate) 630 631 def _generate_fields(self, fields_to_generate):
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc in _generate_fields(self, fields_to_generate) 644 fi = self.pf._get_field_info(*field) 645 try: --> 646 fd = self._generate_field(field) 647 if type(fd) == np.ndarray: 648 fd = self.pf.arr(fd, fi.units)
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc in _generate_field(self, field) 255 tr = self._generate_particle_field(field) 256 else: --> 257 tr = self._generate_fluid_field(field) 258 if tr is None: 259 raise YTCouldNotGenerateField(field, self.pf)
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc in _generate_fluid_field(self, field) 275 rv = self._generate_spatial_fluid(field, ngt_exception.ghost_zones) 276 else: --> 277 rv = finfo(gen_obj) 278 return rv 279
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/fields/derived_field.pyc in __call__(self, data) 176 "for %s" % (self.name,)) 177 with self.unit_registry(data): --> 178 dd = self._function(self, data) 179 for field_name in data.keys(): 180 if field_name not in original_fields:
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/fields/particle_fields.pyc in particle_cic(field, data) 113 def particle_cic(field, data): 114 pos = data[ptype, coord_name] --> 115 d = data.deposit(pos, [data[ptype, mass_name]], method = "cic") 116 d = data.apply_units(d, data[ptype, mass_name].units) 117 d /= data["index", "cell_volume"]
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/octree_subset.pyc in deposit(self, positions, fields, method) 167 mylog.debug("Depositing %s (%s^3) particles into %s Octs", 168 positions.shape[0], positions.shape[0]**0.3333333, nvals[-1]) --> 169 pos = np.asarray(positions.convert_to_units("code_length"), 170 dtype="float64") 171 # We should not need the following if we know in advance all our fields
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/units/yt_array.pyc in convert_to_units(self, units) 366 367 self.units = new_units --> 368 self *= conversion_factor 369 return self 370
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/units/yt_array.pyc in __imul__(self, other) 667 """ See __mul__. """ 668 oth = sanitize_units_mul(self, other) --> 669 return np.multiply(self, oth, out=self) 670 671 def __div__(self, right_object):
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/units/yt_array.pyc in __array_wrap__(self, out_arr, context) 966 # casting to YTArray avoids creating a YTQuantity with size > 1 967 return YTArray(np.array(out_arr, unit)) --> 968 return ret_class(np.array(out_arr), unit) 969 970
MemoryError:
On Sun, Jun 8, 2014 at 10:50 PM, Matthew Turk <matthewturk@gmail.com> wrote:
Hi Brendan,
Hi Matt,
Thanks for your detailed email. Forgive my naivety but why do you need
On Sun, Jun 8, 2014 at 9:21 PM, Brendan Griffen <brendan.f.griffen@gmail.com> wrote: the
oct-tree in the first place? I have a my own fortran code for constructing a cloud in cell mesh and it uses very little overhead (just the n^3 grid and the particle data itself). I then calculate the dx,dy,dzs to the nearest 8 grid points and distribute accordingly in a omp loop which is done in a fraction of a second. Does the situation with yt come about (oct tree etc.) necessarily because of the way it handles particle data? Is it essentially used to map the particles to domains in the grid or something?
That's not naive at all. There are two reasons --
1) The octree is used for indexing for neighbor lookups and early-termination of region selection for particles 2) The octree is used to estimate the "required resolution" for any operation that requires a space-filling value. (i.e., any time that a particle becomes a volume.)
Projections in yt are adaptive, in that they project down to the finest appropriate resolution. There's also the "arbitrary_grid" operation, which does precisely what you're describing, but as it stands right now the octree gets constructed at time of instantiation of the indexing system. Thinking it over, you may be able to avoid that completely by not using load_particles and instead using load_uniform_grid and supplying your desired dimensions. The field names should be the same.
The machine I max memory on has 128GB and the snapshots are using
1024^3
particles. Do you have any idea of how much memory the oct-tree uses as a function of particle/grid number? I am going to try on a 256GB machine (though this is a bit of a hassle). I'll see how I go.
I am disappointed that it's blowing out your RAM. This week I will try to get some memory profiling done. Could you file a bug to this effect, which will help me track it? Peak memory usage during indexing should only be 64 bits * Nparticles, unless you're using load_particles, in which case all the fields will *also* have to be in memory. It's about 8 gigabytes per field. So, I think there's something going wrong.
Thanks.
Brendan
On Sun, Jun 8, 2014 at 6:25 PM, Matthew Turk <matthewturk@gmail.com>
Hi all,
I feel like I owe a brief explanation of why things are tricky right now, what we're planning on doing, and how we're experimenting and developing.
Presently, the particle geometry handlers build a single mesh from all particles in the dataset, along with a coarse bitmask that correlates files to regions in the domain. This requires the allocation of a single int64 array of size Nparticles, which is sorted in place and then fed into an octree construction algorithm that then spits back out the mesh. Each octree component contains 3 64-bit integers and eitehr a void pointer or a pointer to eight other octs. Increasing n_ref decreases the number of octs in this mesh; when smoothing operaitons are conducted, a second "index" mesh is created for looking up particles near mesh points. Mesh points are used for adaptive resolution smoothing and other "deposit particles on the grid somehow" operations (including SPH kernel).
Anyway, because right now it requires a global mesh to be constructed, this is expensive and requires holding a 64-bit integer in memory for each particle. I think if you're loading the particles in differently there is some additional overhead as well, but I'm still a bit surprised you OOM on a 1024^3 dataset.
In general, we don't *need* this global mesh; is can be constructed as required, which would speed up both the initial index phase as well as the final meshing process. I got about 50% of the way to implementing this last fall, but because of various concerns and deadlines I haven't finished it. I intend to get back to it probably in July, right after we put out a 3.0, so that we can have it in time for 3.1. In principle this will make the particle codes much more similar to ARTIO, in that the mesh will be constructed only as required and discarded when no longer required, which will make them much more memory efficient.
But, getting a single mesh for extremely large data is a very high priority; right now for the 10240^3 run we've been loading up individual sub-chunks, which I want to stop doing.
From the technical perspective, these are the things that need to happen on the yt side for particle datasets to move to this "lazy" mode of loading; most of this is based on things learned from 2HOT and ARTIO, and will involve converting to a forest-of-octrees.
* Split into spatially-organized subchunks of ParticleOctreeSubset objects, such that these map 1:Nfiles, and that can be constructed on the fly. * Construct a dual-mesh of the bitmask "ParticleRegion" object that will help with identifying neighbors to a given oct cell, so that if we're inside one octree we know which neighbor octrees to grab if we need particles for smoothing things (fast boundary particle identification is later down the road) * Parallel sort of particles, or using the parallel ring function; may not be necessary after all
All of this is doable, and I'd be happy to work with people if they'd like to take a shot at implementing it, but I've mostly put it on my list for post-3.0.
-Matt
On Sun, Jun 8, 2014 at 2:43 PM, Nathan Goldbaum <
nathan12343@gmail.com>
wrote:
On Sun, Jun 8, 2014 at 12:27 PM, Brendan Griffen <brendan.f.griffen@gmail.com> wrote: > > Also, how do I construct just a zero filled yt array with
dimensions
> (ndim,ndim,ndim)? Thanks
from yt import YTArray from numpy import np
arr = YTArray(np.zeros([ndim, ndim, ndim]), input_units=units_string)
or alternatively:
from yt.units import kiloparsec
arr = kiloparsec*np.zeros([ndim, ndim, ndim])
it doesn't have to be kiloparsec - you can compose the units you want out of any of the unit symbols that live in yt.units.
See this page for a ton more detail about yt's new unit system: http://yt-project.org/docs/dev-3.0/analyzing/units/index.html
> > > Brendan > > > On Sun, Jun 8, 2014 at 3:26 PM, Brendan Griffen > <brendan.f.griffen@gmail.com> wrote: >> >> Hi, >> >> Since I get memory errors. Could I not just read in the blocks of
>> output individually then basically stack the mesh each time. That way >> not >> every single particle of the snapshot has to be loaded at the same >> time. >> Would that just be a case of doing >> >> level = int(math.log(ndim,2)) >> cg = ds.covering_grid(level=level, >> left_edge=[0,0,0],dims=[ndim,ndim,ndim]) >> arr = cg['deposit', 'all_density'] >> arrall += arr >> >> in a loop over each HDF5 block?
It's likely that the memory use is dominated by the octree rather
the covering grid. This is with 1024^3 particles, correct?
You can probably significantly reduce the memory used by the octree by increasing n_ref in the call to load_particles.
See this page for more detail about load_particles:
http://yt-project.org/docs/dev-3.0/examining/loading_data.html#generic-parti...
Larger n_ref means fewer octree cells (lower resolution), but it
also
means lower poisson noise and lower memory use.
Alternatively, as Matt suggested, you could break your 1024^3 ensemble of particles up into chunks, loop over the chunk, creating a particle octree and then a covering grid for each subset of the particles. Your final covering grid is just the sub of the covering grids for each subset of particles.
>> >> >> Thanks. >> Brendan >> >> >> >> >> On Fri, Jun 6, 2014 at 7:26 PM, Matthew Turk < matthewturk@gmail.com> >> wrote: >>> >>> >>> On Jun 6, 2014 4:54 PM, "Brendan Griffen" >>> <brendan.f.griffen@gmail.com> >>> wrote: >>> > >>> > OK great. It is very low resolution but it worked. Thanks for all >>> > your >>> > help. My higher resolution run 1024^3 in 100 Mpc seems to crash on >>> > 128GB >>> > memory machine. I might have to look elsewhere. >>> > >>> >>> If you are looking for low resolution extraction you can tune
>>> memory usage by changing the parameter n_ref to something higher. >>> >>> Supporting extremely large datasets in a single mesh is on the >>> roadmap >>> for the late summer or fall, after a 3.0 release goes out. For now >>> you can >>> also extract before you load in; this is sort of how we are >>> supporting an >>> INCITE project with very large particle counts. >>> >>> >>> > Also, I normally use Canopy distribution but I just use an alias to >>> > loadyt which erases my PYTHONPATH and I can't access scipy and a >>> > few other >>> > libraries any more. What is the best practice here? Should I just >>> > manually >>> > export PYTHONPATH and point to the libraries need in canopy or can >>> > they play >>> > nice together? >>> > >>> > Thanks. >>> > >>> > BG >>> > >>> > >>> > On Fri, Jun 6, 2014 at 2:54 PM, Nathan Goldbaum >>> > <nathan12343@gmail.com> wrote: >>> >> >>> >> >>> >> >>> >> >>> >> On Fri, Jun 6, 2014 at 11:48 AM, Brendan Griffen >>> >> <brendan.f.griffen@gmail.com> wrote: >>> >>> >>> >>> OK great. Thanks. I just wanted a homogeneous mesh. 512^3 with no >>> >>> nesting of any kind. Though when I plot the image it looks
>>> >>> it is >>> >>> assigning particles incorrectly (low resolution on the outside). >>> >>> This is >>> >>> just a test image. >>> >>> >>> >> >>> >> The SlicePlot is visualizing the octree so there is less >>> >> resolution >>> >> where there are fewer particles. If you want to visualize the >>> >> covering grid >>> >> you're going to need to visualize that separately. >>> >> >>> >>> >>> >>> ds = yt.load_particles(data, length_unit=3.08e24, >>> >>> mass_unit=1.9891e33,bbox=bbox) >>> >>> >>> >>> ad = ds.all_data() >>> >>> print ad['deposit', 'all_cic'] >>> >>> slc = yt.SlicePlot(ds, 2, ('deposit', 'all_cic')) >>> >>> slc.set_figure_size(4) >>> >>> cg = ds.covering_grid(level=9, >>> >>> left_edge=[0,0,0],dims=[512,512,512]) >>> >>> >>> >> >>> >> To actually produce the uniform resolution ndarray, you're going >>> >> to >>> >> need to do something like: >>> >> >>> >> array = cg[('deposit', 'all_cic')] >>> >> >>> >> array will then be a 3D array you can do whatever you want with. >>> >> By >>> >> default it has units, but to strip them off you'll just need to >>> >> cast to >>> >> ndarray: >>> >> >>> >> array_without_units = array.v >>> >> >>> >> >>> >>> >>> >>> Also, is there a way to load multiple particle types? >>> >>> >>> >>> Do I just need to stack the particles into the array here? >>> >>> >>> >>> data = {'particle_position_x': pos[:,0], >>> >>> 'particle_position_y': pos[:,1], >>> >>> 'particle_position_z': pos[:,2], >>> >>> 'particle_mass': np.array([mpart]*npart)} >>> >>> >>> >>> Then feed it in as usual? >>> >> >>> >> >>> >> That's right, although if the particle masses are different for >>> >> the >>> >> different particle types that code snippet will need to be >>> >> generalized to >>> >> handle that. >>> >> >>> >> I think in principle it should be possible to make load_particles >>> >> handle different particle types just like an SPH dataset that >>> >> contains >>> >> multiple particle types, but right now that hasn't been >>> >> implemented yet. >>> >> >>> >>> >>> >>> >>> >>> Brendan >>> >>> >>> >>> >>> >>> On Thu, Jun 5, 2014 at 9:44 PM, Nathan Goldbaum >>> >>> <nathan12343@gmail.com> wrote: >>> >>>> >>> >>>> That's right, you can set that via the bbox keyword
wrote: the than the like parameter
>>> >>>> for >>> >>>> load_particles. I'd urge you to take a look at the docstrings >>> >>>> and source >>> >>>> code for load_particles. >>> >>>> >>> >>>> >>> >>>> On Thu, Jun 5, 2014 at 6:34 PM, Brendan Griffen >>> >>>> <brendan.f.griffen@gmail.com> wrote: >>> >>>>> >>> >>>>> Thanks very much Nathan. I tried to load in my own data but I >>> >>>>> think there are too many particles or I have to specifically >>> >>>>> set the domain >>> >>>>> size. >>> >>>>> >>> >>>>> In this area: >>> >>>>> >>> >>>>> data = {'particle_position_x': pos[:,0], >>> >>>>> 'particle_position_y': pos[:,1], >>> >>>>> 'particle_position_z': pos[:,2], >>> >>>>> 'particle_mass': np.array([mpart]*npart)} >>> >>>>> >>> >>>>> ds = yt.load_particles(data, length_unit=3.08e24, >>> >>>>> mass_unit=1.9891e36) >>> >>>>> ad = ds.all_data() >>> >>>>> print ad['deposit', 'all_cic'] >>> >>>>> >>> >>>>> In [3]: run ytcic.py >>> >>>>> yt : [INFO ] 2014-06-05 21:29:06,183 Parameters: >>> >>>>> current_time >>> >>>>> = 0.0 >>> >>>>> yt : [INFO ] 2014-06-05 21:29:06,183 Parameters: >>> >>>>> domain_dimensions = [2 2 2] >>> >>>>> yt : [INFO ] 2014-06-05 21:29:06,184 Parameters: >>> >>>>> domain_left_edge = [ 0. 0. 0.] >>> >>>>> yt : [INFO ] 2014-06-05 21:29:06,185 Parameters: >>> >>>>> domain_right_edge = [ 1. 1. 1.] >>> >>>>> yt : [INFO ] 2014-06-05 21:29:06,185 Parameters: >>> >>>>> cosmological_simulation = 0.0 >>> >>>>> yt : [INFO ] 2014-06-05 21:29:06,188 Allocating for >>> >>>>> 1.342e+08 >>> >>>>> particles >>> >>>>> >>> >>>>> >>> >>>>>
>>> >>>>> YTDomainOverflow Traceback (most >>> >>>>> recent >>> >>>>> call last) >>> >>>>> >>> >>>>> >>> >>>>> /nfs/blank/h4231/bgriffen/data/lib/yt-x86_64/lib/python2.7/site-packages/IPython/utils/py3compat.pyc >>> >>>>> in execfile(fname, *where) >>> >>>>> 202 else: >>> >>>>> 203 filename = fname >>> >>>>> --> 204 __builtin__.execfile(filename, *where) >>> >>>>> >>> >>>>> >>> >>>>> >>> >>>>> /nfs/blank/h4231/bgriffen/work/projects/caterpillar/c2ray/cic/ytcic.py in >>> >>>>> <module>() >>> >>>>> 52 >>> >>>>> 53 ad = ds.all_data() >>> >>>>> ---> 54 print ad['deposit', 'all_cic'] >>> >>>>> 55 slc = yt.SlicePlot(ds, 2, ('deposit', 'all_cic')) >>> >>>>> 56 slc.set_figure_size(4) >>> >>>>> >>> >>>>> >>> >>>>> >>> >>>>> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc >>> >>>>> in __getitem__(self, key) >>> >>>>> 205 Returns a single field. Will add if necessary. >>> >>>>> 206 """ >>> >>>>> --> 207 f = self._determine_fields([key])[0] >>> >>>>> 208 if f not in self.field_data and key not in >>> >>>>> self.field_data: >>> >>>>> 209 if f in self._container_fields: >>> >>>>> >>> >>>>> >>> >>>>> >>> >>>>> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc >>> >>>>> in _determine_fields(self, fields) >>> >>>>> 453 raise YTFieldNotParseable(field) >>> >>>>> 454 ftype, fname = field >>> >>>>> --> 455 finfo = self.pf._get_field_info(ftype, >>> >>>>> fname) >>> >>>>> 456 else: >>> >>>>> 457 fname = field >>> >>>>> >>> >>>>> >>> >>>>> >>> >>>>> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/static_output.pyc >>> >>>>> in _get_field_info(self, ftype, fname) >>> >>>>> 445 _last_finfo = None >>> >>>>> 446 def _get_field_info(self, ftype, fname = None): >>> >>>>> --> 447 self.index >>> >>>>> 448 if fname is None: >>> >>>>> 449 ftype, fname = "unknown", ftype >>> >>>>> >>> >>>>> >>> >>>>> >>> >>>>> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/static_output.pyc >>> >>>>> in index(self) >>> >>>>> 277 raise RuntimeError("You should not >>> >>>>> instantiate Dataset.") >>> >>>>> 278 self._instantiated_index = >>> >>>>> self._index_class( >>> >>>>> --> 279 self, dataset_type=self.dataset_type) >>> >>>>> 280 # Now we do things that we need an >>> >>>>> instantiated index for >>> >>>>> 281 # ...first off, we create our field_info >>> >>>>> now. >>> >>>>> >>> >>>>> >>> >>>>> >>> >>>>> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/frontends/stream/data_structures.pyc >>> >>>>> in __init__(self, pf, dataset_type) >>> >>>>> 942 def __init__(self, pf, dataset_type = None): >>> >>>>> 943 self.stream_handler = pf.stream_handler >>> >>>>> --> 944 super(StreamParticleIndex, self).__init__(pf, >>> >>>>> dataset_type) >>> >>>>> 945 >>> >>>>> 946 def _setup_data_io(self): >>> >>>>> >>> >>>>> >>> >>>>> >>> >>>>> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/geometry/particle_geometry_handler.pyc >>> >>>>> in __init__(self, pf, dataset_type) >>> >>>>> 48 self.directory = >>> >>>>> os.path.dirname(self.index_filename) >>> >>>>> 49 self.float_type = np.float64 >>> >>>>> ---> 50 super(ParticleIndex, self).__init__(pf, >>> >>>>> dataset_type) >>> >>>>> 51 >>> >>>>> 52 def _setup_geometry(self): >>> >>>>> >>> >>>>> >>> >>>>> >>> >>>>> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/geometry/geometry_handler.pyc >>> >>>>> in __init__(self, pf, dataset_type) >>> >>>>> 54 >>> >>>>> 55 mylog.debug("Setting up domain geometry.") >>> >>>>> ---> 56 self._setup_geometry() >>> >>>>> 57 >>> >>>>> 58 mylog.debug("Initializing data grid data IO") >>> >>>>> >>> >>>>> >>> >>>>> >>> >>>>> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/geometry/particle_geometry_handler.pyc >>> >>>>> in _setup_geometry(self) >>> >>>>> 52 def _setup_geometry(self): >>> >>>>> 53 mylog.debug("Initializing Particle Geometry >>> >>>>> Handler.") >>> >>>>> ---> 54 self._initialize_particle_handler() >>> >>>>> 55 >>> >>>>> 56 >>> >>>>> >>> >>>>> >>> >>>>> >>> >>>>> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/geometry/particle_geometry_handler.pyc >>> >>>>> in _initialize_particle_handler(self) >>> >>>>> 87 pf.domain_left_edge, >>> >>>>> pf.domain_right_edge, >>> >>>>> 88 [N, N, N], len(self.data_files)) >>> >>>>> ---> 89 self._initialize_indices() >>> >>>>> 90 self.oct_handler.finalize() >>> >>>>> 91 self.max_level = self.oct_handler.max_level >>> >>>>> >>> >>>>> >>> >>>>> >>> >>>>> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/geometry/particle_geometry_handler.pyc >>> >>>>> in _initialize_indices(self) >>> >>>>> 109 npart = >>> >>>>> sum(data_file.total_particles.values()) >>> >>>>> 110 morton[ind:ind + npart] = \ >>> >>>>> --> 111 self.io._initialize_index(data_file, >>> >>>>> self.regions) >>> >>>>> 112 ind += npart >>> >>>>> 113 morton.sort() >>> >>>>> >>> >>>>> >>> >>>>> >>> >>>>> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/frontends/stream/io.pyc in >>> >>>>> _initialize_index(self, data_file, regions) >>> >>>>> 144 raise YTDomainOverflow(pos.min(axis=0), >>> >>>>> pos.max(axis=0), >>> >>>>> 145 >>> >>>>> data_file.pf.domain_left_edge, >>> >>>>> --> 146 >>> >>>>> data_file.pf.domain_right_edge) >>> >>>>> 147 regions.add_data_file(pos, >>> >>>>> data_file.file_id) >>> >>>>> 148 morton.append(compute_morton( >>> >>>>> >>> >>>>> YTDomainOverflow: Particle bounds [ 0. 0. 0.] and [ >>> >>>>> 99.99999237 >>> >>>>> 99.99999237 99.99999237] exceed domain bounds [ 0. 0. 0.] >>> >>>>> code_length and >>> >>>>> [ 1. 1. 1.] code_length >>> >>>>> >>> >>>>> >>> >>>>> >>> >>>>> On Thu, Jun 5, 2014 at 8:22 PM, Nathan Goldbaum >>> >>>>> <nathan12343@gmail.com> wrote: >>> >>>>>> >>> >>>>>> Here's a worked out example that does what you're looking for >>> >>>>>> using a fake 1 million particle dataset: >>> >>>>>> >>> >>>>>> >>> >>>>>> http://nbviewer.ipython.org/gist/ngoldbaum/546d37869aafe71cfe38 >>> >>>>>> >>> >>>>>> In this notebook I make use of two key yt features: >>> >>>>>> `load_particles`, and `covering_grid`. >>> >>>>>> >>> >>>>>> load_particles creates a "stream" dataset based on in-memory >>> >>>>>> data >>> >>>>>> fed in as a numpy array. This dataset acts just like an >>> >>>>>> on-disk simulation >>> >>>>>> dataset, but doesn't come with the baggage of needing to write >>> >>>>>> a custom >>> >>>>>> frontend to read a specific data format off disk. >>> >>>>>> >>> >>>>>> covering_grid is a way to generate uniform resolution data >>> >>>>>> from >>> >>>>>> an AMR dataset. It acts like a python dictionary where the >>> >>>>>> keys are field >>> >>>>>> names and returns 3D numpy arrays of whatever uniform >>> >>>>>> resolution you specify >>> >>>>>> when you create the covering_grid. >>> >>>>>> >>> >>>>>> Note that if you're using load_particles all of your data >>> >>>>>> needs >>> >>>>>> to live in memory. If your data is too big for that you'll >>> >>>>>> need to write a >>> >>>>>> frontend for your data format or use a memmap to an on-disk >>> >>>>>> file somehow. >>> >>>>>> I'm not an expert on that but others on the list should be >>> >>>>>> able to help out. >>> >>>>>> >>> >>>>>> Hope that gets you well on your way :) >>> >>>>>> >>> >>>>>> -Nathan >>> >>>>>> >>> >>>>>> >>> >>>>>> On Thu, Jun 5, 2014 at 5:04 PM, Desika Narayanan >>> >>>>>> <dnarayan@haverford.edu> wrote: >>> >>>>>>> >>> >>>>>>> Hey Brendan, >>> >>>>>>> >>> >>>>>>> A couple of extra tools you might find helpful in conjunction >>> >>>>>>> with Nathan's example of depositing the particles onto an >>> >>>>>>> octree are at: >>> >>>>>>> >>> >>>>>>> http://paste.yt-project.org/show/4737/ >>> >>>>>>> >>> >>>>>>> Where I load a gadget snapshot, and then recover the >>> >>>>>>> coordinates >>> >>>>>>> and width of each cell. >>> >>>>>>> >>> >>>>>>> In response to your last question - the particles are >>> >>>>>>> deposited >>> >>>>>>> into an octree grid (so, you'll see that the cell sizes >>> >>>>>>> aren't all the same >>> >>>>>>> size). I don't know if depositing onto a regular NxNxN mesh >>> >>>>>>> is possible, >>> >>>>>>> though would be interested to hear if so. >>> >>>>>>> >>> >>>>>>> -d >>> >>>>>>> >>> >>>>>>> >>> >>>>>>> On Thu, Jun 5, 2014 at 7:58 PM, Brendan Griffen >>> >>>>>>> <brendan.f.griffen@gmail.com> wrote: >>> >>>>>>>> >>> >>>>>>>> Thanks. I'll get the "bleeding edge" version first then try >>> >>>>>>>> your suggestions. Though I want to return the NxNxN array >>> >>>>>>>> and be able to >>> >>>>>>>> write this mesh to a file. It is *only* using the cic part >>> >>>>>>>> of yt and it >>> >>>>>>>> should return the mesh to be written? Just wanted to >>> >>>>>>>> clarify? >>> >>>>>>>> >>> >>>>>>>> Thanks. >>> >>>>>>>> Brendan >>> >>>>>>>> >>> >>>>>>>> >>> >>>>>>>> On Thu, Jun 5, 2014 at 6:49 PM, Nathan Goldbaum >>> >>>>>>>> <nathan12343@gmail.com> wrote: >>> >>>>>>>>> >>> >>>>>>>>> >>> >>>>>>>>> >>> >>>>>>>>> >>> >>>>>>>>> On Thu, Jun 5, 2014 at 3:36 PM, John ZuHone >>> >>>>>>>>> <jzuhone@gmail.com> wrote: >>> >>>>>>>>>> >>> >>>>>>>>>> Hi Brendan, >>> >>>>>>>>>> >>> >>>>>>>>>> Which version of yt are you using? >>> >>>>>>>>>> >>> >>>>>>>>>> If you're using 3.0, this is actually fairly easy. If you >>> >>>>>>>>>> look in yt.fields.particle_fields.py, around line 85, you >>> >>>>>>>>>> can see how this >>> >>>>>>>>>> is done for the "particle_density" and "particle_mass" >>> >>>>>>>>>> fields. Basically you >>> >>>>>>>>>> can call a "deposit" method which takes the particle field >>> >>>>>>>>>> quantity you want >>> >>>>>>>>>> deposited and deposits it into cells. The underlying >>> >>>>>>>>>> calculation is done >>> >>>>>>>>>> using Cython, so it's fast. >>> >>>>>>>>> >>> >>>>>>>>> >>> >>>>>>>>> And you shouldn't ever actually need to call these >>> >>>>>>>>> "deposit" >>> >>>>>>>>> functions, since "deposit" is exposed as a field type for >>> >>>>>>>>> all datasets that >>> >>>>>>>>> contain particles. >>> >>>>>>>>> >>> >>>>>>>>> Here is a notebook that does this for Enzo AMR data: >>> >>>>>>>>> >>> >>>>>>>>> >>> >>>>>>>>> >>> >>>>>>>>> http://nbviewer.ipython.org/gist/ngoldbaum/5e19e4e6cc2bf330149c >>> >>>>>>>>> >>> >>>>>>>>> This dataset contains about a million particles and >>> >>>>>>>>> generates >>> >>>>>>>>> a CIC deposition for the whole domain in about 6 seconds >>> >>>>>>>>> from a cold start. >>> >>>>>>>>> >>> >>>>>>>>>> >>> >>>>>>>>>> >>> >>>>>>>>>> If you're using 2.x, then you can do the same thing, but >>> >>>>>>>>>> it's >>> >>>>>>>>>> not as straightforward. You can see how this works in >>> >>>>>>>>>> yt.data_objects.universal_fields.py, around line 986, >>> >>>>>>>>>> where the >>> >>>>>>>>>> "particle_density" field is defined. Basically, it calls >>> >>>>>>>>>> CICDeposit_3, which >>> >>>>>>>>>> is in yt.utilities.lib.CICDeposit.pyx. >>> >>>>>>>>>> >>> >>>>>>>>>> Let me know if you need any more clarification. >>> >>>>>>>>>> >>> >>>>>>>>>> Best, >>> >>>>>>>>>> >>> >>>>>>>>>> John Z >>> >>>>>>>>>> >>> >>>>>>>>>> On Jun 5, 2014, at 6:07 PM, Brendan Griffen >>> >>>>>>>>>> <brendan.f.griffen@gmail.com> wrote: >>> >>>>>>>>>> >>> >>>>>>>>>> > Hi, >>> >>>>>>>>>> > >>> >>>>>>>>>> > I was wondering if there were any Cython routines within >>> >>>>>>>>>> > yt >>> >>>>>>>>>> > which takes particle data and converts it into a >>> >>>>>>>>>> > cloud-in-cell based mesh >>> >>>>>>>>>> > which can be written to a file of my choosing. >>> >>>>>>>>> >>> >>>>>>>>> >>> >>>>>>>>> What sort of mesh were you looking for? yt will internally >>> >>>>>>>>> construct an octree if it is fed particle data. I'm not >>> >>>>>>>>> sure whether this >>> >>>>>>>>> octree can be saved to disk for later analysis. >>> >>>>>>>>> >>> >>>>>>>>> It's also possible to create a uniform resolution covering >>> >>>>>>>>> grid containing field data for a deposited quantity, which >>> >>>>>>>>> can be quite >>> >>>>>>>>> easily saved to disk in a number of ways. >>> >>>>>>>>> >>> >>>>>>>>>> >>> >>>>>>>>>> I heard a while ago there was some such functionality but >>> >>>>>>>>>> it >>> >>>>>>>>>> could be too far down the yt rabbit hole to be used as a >>> >>>>>>>>>> standalone? Is this >>> >>>>>>>>>> true? I have my own Python code for doing it but it just >>> >>>>>>>>>> isn't fast enough >>> >>>>>>>>>> and thought I'd ask the yt community if there were any >>> >>>>>>>>>> wrapper tools >>> >>>>>>>>>> available to boost the speed. >>> >>>>>>>>>> > >>> >>>>>>>>>> > Thanks. >>> >>>>>>>>>> > Brendan >>> >>>>>>>>>> > _______________________________________________ >>> >>>>>>>>>> > yt-users mailing list >>> >>>>>>>>>> > yt-users@lists.spacepope.org >>> >>>>>>>>>> > >>> >>>>>>>>>> > >>> >>>>>>>>>> > http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >>> >>>>>>>>>> >>> >>>>>>>>>> _______________________________________________ >>> >>>>>>>>>> yt-users mailing list >>> >>>>>>>>>> yt-users@lists.spacepope.org >>> >>>>>>>>>> >>> >>>>>>>>>> >>> >>>>>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >>> >>>>>>>>> >>> >>>>>>>>> >>> >>>>>>>>> >>> >>>>>>>>> _______________________________________________ >>> >>>>>>>>> yt-users mailing list >>> >>>>>>>>> yt-users@lists.spacepope.org >>> >>>>>>>>> >>> >>>>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >>> >>>>>>>>> >>> >>>>>>>> >>> >>>>>>>> >>> >>>>>>>> _______________________________________________ >>> >>>>>>>> yt-users mailing list >>> >>>>>>>> yt-users@lists.spacepope.org >>> >>>>>>>> >>> >>>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >>> >>>>>>>> >>> >>>>>>> >>> >>>>>>> >>> >>>>>>> _______________________________________________ >>> >>>>>>> yt-users mailing list >>> >>>>>>> yt-users@lists.spacepope.org >>> >>>>>>> >>> >>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >>> >>>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> _______________________________________________ >>> >>>>>> yt-users mailing list >>> >>>>>> yt-users@lists.spacepope.org >>> >>>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >>> >>>>>> >>> >>>>> >>> >>>>> >>> >>>>> _______________________________________________ >>> >>>>> yt-users mailing list >>> >>>>> yt-users@lists.spacepope.org >>> >>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >>> >>>>> >>> >>>> >>> >>>> >>> >>>> _______________________________________________ >>> >>>> yt-users mailing list >>> >>>> yt-users@lists.spacepope.org >>> >>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >>> >>>> >>> >>> >>> >>> >>> >>> _______________________________________________ >>> >>> yt-users mailing list >>> >>> yt-users@lists.spacepope.org >>> >>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >>> >>> >>> >> >>> >> >>> >> _______________________________________________ >>> >> yt-users mailing list >>> >> yt-users@lists.spacepope.org >>> >> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >>> >> >>> > >>> > >>> > _______________________________________________ >>> > yt-users mailing list >>> > yt-users@lists.spacepope.org >>> > http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >>> > >>> >>> >>> _______________________________________________ >>> yt-users mailing list >>> yt-users@lists.spacepope.org >>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >>> >> > > > _______________________________________________ > yt-users mailing list > yt-users@lists.spacepope.org > http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
On Mon, Jun 9, 2014 at 12:32 AM, Brendan Griffen < brendan.f.griffen@gmail.com> wrote:
Thanks Nathan. Comparing the two though:
26 1536.664 MiB 1130.480 MiB uniform_array = grid_object['deposit', 'all_cic']
24 1285.703 MiB 890.539 MiB cic_density = ad["deposit", "all_cic"]
Isn't it the uniform grid which uses the most memory? I'm running the profiling now and will get back to you. I hope in spite of the fact that it does crash it will still give me some useful output.
That's true - I was thinking it might be more memory efficient since there is no need to construct the covering grid in addition to the octree.
Brendan
On Mon, Jun 9, 2014 at 2:05 AM, Nathan Goldbaum <nathan12343@gmail.com> wrote:
Hey Brendan,
Could you try running your script using the memory_profiler module on pypi?
Here's an example script that uses the memory profiler: http://paste.yt-project.org/show/4748/
and the output for that script: http://paste.yt-project.org/show/4749/
For what it's worth, it does indeed look like matt's suggestion to use load_uniform_grid is an option, and might be more memory efficient in the end since you will go directly to the uniform grid you want without creating an octree. Here's an example: http://paste.yt-project.org/show/4750/
Here's the memory usage information for that example: http://paste.yt-project.org/show/4751/
I used a 256^3 uniform grid with normally distributed random data - I'm not sure whether it will also be more memory efficient in your case.
On Sun, Jun 8, 2014 at 9:32 PM, Brendan Griffen < brendan.f.griffen@gmail.com> wrote:
This is the full error if it helps at all? It is indeed, loading in all of the quantities.
Loading particles... --> Loading particle type: 1 yt : [INFO ] 2014-06-08 15:12:05,540 Parameters: current_time = 0.0 yt : [INFO ] 2014-06-08 15:12:05,540 Parameters: domain_dimensions = [2 2 2] yt : [INFO ] 2014-06-08 15:12:05,541 Parameters: domain_left_edge = [ 0. 0. 0.] yt : [INFO ] 2014-06-08 15:12:05,542 Parameters: domain_right_edge = [ 100. 100. 100.] yt : [INFO ] 2014-06-08 15:12:05,542 Parameters: cosmological_simulation = 0.0 yt : [INFO ] 2014-06-08 15:12:05,548 Allocating for 1.074e+09 particles yt : [INFO ] 2014-06-08 15:16:16,195 Identified 7.584e+07 octs yt : [INFO ] 2014-06-08 15:16:16,299 Loading field plugins. yt : [INFO ] 2014-06-08 15:16:16,299 Loaded angular_momentum (8 new fields) yt : [INFO ] 2014-06-08 15:16:16,299 Loaded astro (14 new fields) yt : [INFO ] 2014-06-08 15:16:16,300 Loaded cosmology (20 new fields) yt : [INFO ] 2014-06-08 15:16:16,300 Loaded fluid (56 new fields) yt : [INFO ] 2014-06-08 15:16:16,301 Loaded fluid_vector (88 new fields) yt : [INFO ] 2014-06-08 15:16:16,301 Loaded geometric (103 new fields) yt : [INFO ] 2014-06-08 15:16:16,301 Loaded local (103 new fields) yt : [INFO ] 2014-06-08 15:16:16,302 Loaded magnetic_field (109 new fields) yt : [INFO ] 2014-06-08 15:16:16,302 Loaded species (109 new fields)
--------------------------------------------------------------------------- MemoryError Traceback (most recent call last) /nfs/blank/h4231/bgriffen/data/lib/yt-x86_64/lib/python2.7/site-packages/IPython/utils/py3compat.pyc in execfile(fname, *where) 202 else: 203 filename = fname --> 204 __builtin__.execfile(filename, *where)
/nfs/blank/h4231/bgriffen/work/projects/caterpillar/c2ray/cic/ytcic.py in <module>() 99 slc.set_figure_size(4) 100 slc.save() --> 101 102 for ndim in dimlist: 103 print
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc in __getitem__(self, key) 218 return self.field_data[f] 219 else: --> 220 self.get_data(f) 221 # fi.units is the unit expression string. We depend on the registry 222 # hanging off the dataset to define this unit object.
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc in get_data(self, fields) 627 628 fields_to_generate += gen_fluids + gen_particles --> 629 self._generate_fields(fields_to_generate) 630 631 def _generate_fields(self, fields_to_generate):
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc in _generate_fields(self, fields_to_generate) 644 fi = self.pf._get_field_info(*field) 645 try: --> 646 fd = self._generate_field(field) 647 if type(fd) == np.ndarray: 648 fd = self.pf.arr(fd, fi.units)
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc in _generate_field(self, field) 255 tr = self._generate_particle_field(field) 256 else: --> 257 tr = self._generate_fluid_field(field) 258 if tr is None: 259 raise YTCouldNotGenerateField(field, self.pf)
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc in _generate_fluid_field(self, field) 273 finfo.check_available(gen_obj) 274 except NeedsGridType as ngt_exception: --> 275 rv = self._generate_spatial_fluid(field, ngt_exception.ghost_zones) 276 else: 277 rv = finfo(gen_obj)
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc in _generate_spatial_fluid(self, field, ngz) 289 o = self._current_chunk.objs[0] 290 with o._activate_cache(): --> 291 ind += o.select(self.selector, self[field], rv, ind) 292 else: 293 chunks = self.index._chunk(self, "spatial", ngz = ngz)
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc in __getitem__(self, key) 218 return self.field_data[f] 219 else: --> 220 self.get_data(f) 221 # fi.units is the unit expression string. We depend on the registry 222 # hanging off the dataset to define this unit object.
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc in get_data(self, fields) 627 628 fields_to_generate += gen_fluids + gen_particles --> 629 self._generate_fields(fields_to_generate) 630 631 def _generate_fields(self, fields_to_generate):
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc in _generate_fields(self, fields_to_generate) 644 fi = self.pf._get_field_info(*field) 645 try: --> 646 fd = self._generate_field(field) 647 if type(fd) == np.ndarray: 648 fd = self.pf.arr(fd, fi.units)
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc in _generate_field(self, field) 255 tr = self._generate_particle_field(field) 256 else: --> 257 tr = self._generate_fluid_field(field) 258 if tr is None: 259 raise YTCouldNotGenerateField(field, self.pf)
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc in _generate_fluid_field(self, field) 275 rv = self._generate_spatial_fluid(field, ngt_exception.ghost_zones) 276 else: --> 277 rv = finfo(gen_obj) 278 return rv 279
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/fields/derived_field.pyc in __call__(self, data) 176 "for %s" % (self.name,)) 177 with self.unit_registry(data): --> 178 dd = self._function(self, data) 179 for field_name in data.keys(): 180 if field_name not in original_fields:
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/fields/particle_fields.pyc in particle_cic(field, data) 113 def particle_cic(field, data): 114 pos = data[ptype, coord_name] --> 115 d = data.deposit(pos, [data[ptype, mass_name]], method = "cic") 116 d = data.apply_units(d, data[ptype, mass_name].units) 117 d /= data["index", "cell_volume"]
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/octree_subset.pyc in deposit(self, positions, fields, method) 167 mylog.debug("Depositing %s (%s^3) particles into %s Octs", 168 positions.shape[0], positions.shape[0]**0.3333333, nvals[-1]) --> 169 pos = np.asarray(positions.convert_to_units("code_length"), 170 dtype="float64") 171 # We should not need the following if we know in advance all our fields
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/units/yt_array.pyc in convert_to_units(self, units) 366 367 self.units = new_units --> 368 self *= conversion_factor 369 return self 370
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/units/yt_array.pyc in __imul__(self, other) 667 """ See __mul__. """ 668 oth = sanitize_units_mul(self, other) --> 669 return np.multiply(self, oth, out=self) 670 671 def __div__(self, right_object):
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/units/yt_array.pyc in __array_wrap__(self, out_arr, context) 966 # casting to YTArray avoids creating a YTQuantity with size > 1 967 return YTArray(np.array(out_arr, unit)) --> 968 return ret_class(np.array(out_arr), unit) 969 970
MemoryError:
On Sun, Jun 8, 2014 at 10:50 PM, Matthew Turk <matthewturk@gmail.com> wrote:
Hi Brendan,
On Sun, Jun 8, 2014 at 9:21 PM, Brendan Griffen <brendan.f.griffen@gmail.com> wrote:
Hi Matt,
Thanks for your detailed email. Forgive my naivety but why do you need the oct-tree in the first place? I have a my own fortran code for constructing a cloud in cell mesh and it uses very little overhead (just the n^3 grid and the particle data itself). I then calculate the dx,dy,dzs to the nearest 8 grid points and distribute accordingly in a omp loop which is done in a fraction of a second. Does the situation with yt come about (oct tree etc.) necessarily because of the way it handles particle data? Is it essentially used to map the particles to domains in the grid or something?
That's not naive at all. There are two reasons --
1) The octree is used for indexing for neighbor lookups and early-termination of region selection for particles 2) The octree is used to estimate the "required resolution" for any operation that requires a space-filling value. (i.e., any time that a particle becomes a volume.)
Projections in yt are adaptive, in that they project down to the finest appropriate resolution. There's also the "arbitrary_grid" operation, which does precisely what you're describing, but as it stands right now the octree gets constructed at time of instantiation of the indexing system. Thinking it over, you may be able to avoid that completely by not using load_particles and instead using load_uniform_grid and supplying your desired dimensions. The field names should be the same.
The machine I max memory on has 128GB and the snapshots are using
1024^3
particles. Do you have any idea of how much memory the oct-tree uses as a function of particle/grid number? I am going to try on a 256GB machine (though this is a bit of a hassle). I'll see how I go.
I am disappointed that it's blowing out your RAM. This week I will try to get some memory profiling done. Could you file a bug to this effect, which will help me track it? Peak memory usage during indexing should only be 64 bits * Nparticles, unless you're using load_particles, in which case all the fields will *also* have to be in memory. It's about 8 gigabytes per field. So, I think there's something going wrong.
Thanks.
Brendan
On Sun, Jun 8, 2014 at 6:25 PM, Matthew Turk <matthewturk@gmail.com>
Hi all,
I feel like I owe a brief explanation of why things are tricky right now, what we're planning on doing, and how we're experimenting and developing.
Presently, the particle geometry handlers build a single mesh from
all
particles in the dataset, along with a coarse bitmask that correlates files to regions in the domain. This requires the allocation of a single int64 array of size Nparticles, which is sorted in place and then fed into an octree construction algorithm that then spits back out the mesh. Each octree component contains 3 64-bit integers and eitehr a void pointer or a pointer to eight other octs. Increasing n_ref decreases the number of octs in this mesh; when smoothing operaitons are conducted, a second "index" mesh is created for looking up particles near mesh points. Mesh points are used for adaptive resolution smoothing and other "deposit particles on the grid somehow" operations (including SPH kernel).
Anyway, because right now it requires a global mesh to be constructed, this is expensive and requires holding a 64-bit integer in memory for each particle. I think if you're loading the particles in differently there is some additional overhead as well, but I'm still a bit surprised you OOM on a 1024^3 dataset.
In general, we don't *need* this global mesh; is can be constructed as required, which would speed up both the initial index phase as well as the final meshing process. I got about 50% of the way to implementing this last fall, but because of various concerns and deadlines I haven't finished it. I intend to get back to it probably in July, right after we put out a 3.0, so that we can have it in time for 3.1. In principle this will make the particle codes much more similar to ARTIO, in that the mesh will be constructed only as required and discarded when no longer required, which will make them much more memory efficient.
But, getting a single mesh for extremely large data is a very high priority; right now for the 10240^3 run we've been loading up individual sub-chunks, which I want to stop doing.
From the technical perspective, these are the things that need to happen on the yt side for particle datasets to move to this "lazy" mode of loading; most of this is based on things learned from 2HOT and ARTIO, and will involve converting to a forest-of-octrees.
* Split into spatially-organized subchunks of ParticleOctreeSubset objects, such that these map 1:Nfiles, and that can be constructed on the fly. * Construct a dual-mesh of the bitmask "ParticleRegion" object that will help with identifying neighbors to a given oct cell, so that if we're inside one octree we know which neighbor octrees to grab if we need particles for smoothing things (fast boundary particle identification is later down the road) * Parallel sort of particles, or using the parallel ring function; may not be necessary after all
All of this is doable, and I'd be happy to work with people if they'd like to take a shot at implementing it, but I've mostly put it on my list for post-3.0.
-Matt
On Sun, Jun 8, 2014 at 2:43 PM, Nathan Goldbaum < nathan12343@gmail.com> wrote: > > > > On Sun, Jun 8, 2014 at 12:27 PM, Brendan Griffen > <brendan.f.griffen@gmail.com> wrote: >> >> Also, how do I construct just a zero filled yt array with dimensions >> (ndim,ndim,ndim)? Thanks > > > > from yt import YTArray > from numpy import np > > arr = YTArray(np.zeros([ndim, ndim, ndim]), input_units=units_string) > > or alternatively: > > from yt.units import kiloparsec > > arr = kiloparsec*np.zeros([ndim, ndim, ndim]) > > it doesn't have to be kiloparsec - you can compose the units you want > out of > any of the unit symbols that live in yt.units. > > See this page for a ton more detail about yt's new unit system: > http://yt-project.org/docs/dev-3.0/analyzing/units/index.html > >> >> >> Brendan >> >> >> On Sun, Jun 8, 2014 at 3:26 PM, Brendan Griffen >> <brendan.f.griffen@gmail.com> wrote: >>> >>> Hi, >>> >>> Since I get memory errors. Could I not just read in the blocks of the >>> output individually then basically stack the mesh each time. That way >>> not >>> every single particle of the snapshot has to be loaded at the same >>> time. >>> Would that just be a case of doing >>> >>> level = int(math.log(ndim,2)) >>> cg = ds.covering_grid(level=level, >>> left_edge=[0,0,0],dims=[ndim,ndim,ndim]) >>> arr = cg['deposit', 'all_density'] >>> arrall += arr >>> >>> in a loop over each HDF5 block? > > > It's likely that the memory use is dominated by the octree rather
> the > covering grid. This is with 1024^3 particles, correct? > > You can probably significantly reduce the memory used by the octree by > increasing n_ref in the call to load_particles. > > See this page for more detail about load_particles: > > http://yt-project.org/docs/dev-3.0/examining/loading_data.html#generic-parti... > > Larger n_ref means fewer octree cells (lower resolution), but it also > means > lower poisson noise and lower memory use. > > Alternatively, as Matt suggested, you could break your 1024^3 ensemble > of > particles up into chunks, loop over the chunk, creating a particle > octree > and then a covering grid for each subset of the particles. Your final > covering grid is just the sub of the covering grids for each subset of > particles. > >>> >>> >>> Thanks. >>> Brendan >>> >>> >>> >>> >>> On Fri, Jun 6, 2014 at 7:26 PM, Matthew Turk < matthewturk@gmail.com> >>> wrote: >>>> >>>> >>>> On Jun 6, 2014 4:54 PM, "Brendan Griffen" >>>> <brendan.f.griffen@gmail.com> >>>> wrote: >>>> > >>>> > OK great. It is very low resolution but it worked. Thanks for all >>>> > your >>>> > help. My higher resolution run 1024^3 in 100 Mpc seems to crash on >>>> > 128GB >>>> > memory machine. I might have to look elsewhere. >>>> > >>>> >>>> If you are looking for low resolution extraction you can tune
>>>> memory usage by changing the parameter n_ref to something higher. >>>> >>>> Supporting extremely large datasets in a single mesh is on the >>>> roadmap >>>> for the late summer or fall, after a 3.0 release goes out. For now >>>> you can >>>> also extract before you load in; this is sort of how we are >>>> supporting an >>>> INCITE project with very large particle counts. >>>> >>>> >>>> > Also, I normally use Canopy distribution but I just use an alias to >>>> > loadyt which erases my PYTHONPATH and I can't access scipy and a >>>> > few other >>>> > libraries any more. What is the best practice here? Should I just >>>> > manually >>>> > export PYTHONPATH and point to the libraries need in canopy or can >>>> > they play >>>> > nice together? >>>> > >>>> > Thanks. >>>> > >>>> > BG >>>> > >>>> > >>>> > On Fri, Jun 6, 2014 at 2:54 PM, Nathan Goldbaum >>>> > <nathan12343@gmail.com> wrote: >>>> >> >>>> >> >>>> >> >>>> >> >>>> >> On Fri, Jun 6, 2014 at 11:48 AM, Brendan Griffen >>>> >> <brendan.f.griffen@gmail.com> wrote: >>>> >>> >>>> >>> OK great. Thanks. I just wanted a homogeneous mesh. 512^3 with no >>>> >>> nesting of any kind. Though when I plot the image it looks
>>>> >>> it is >>>> >>> assigning particles incorrectly (low resolution on the outside). >>>> >>> This is >>>> >>> just a test image. >>>> >>> >>>> >> >>>> >> The SlicePlot is visualizing the octree so there is less >>>> >> resolution >>>> >> where there are fewer particles. If you want to visualize the >>>> >> covering grid >>>> >> you're going to need to visualize that separately. >>>> >> >>>> >>> >>>> >>> ds = yt.load_particles(data, length_unit=3.08e24, >>>> >>> mass_unit=1.9891e33,bbox=bbox) >>>> >>> >>>> >>> ad = ds.all_data() >>>> >>> print ad['deposit', 'all_cic'] >>>> >>> slc = yt.SlicePlot(ds, 2, ('deposit', 'all_cic')) >>>> >>> slc.set_figure_size(4) >>>> >>> cg = ds.covering_grid(level=9, >>>> >>> left_edge=[0,0,0],dims=[512,512,512]) >>>> >>> >>>> >> >>>> >> To actually produce the uniform resolution ndarray, you're going >>>> >> to >>>> >> need to do something like: >>>> >> >>>> >> array = cg[('deposit', 'all_cic')] >>>> >> >>>> >> array will then be a 3D array you can do whatever you want with. >>>> >> By >>>> >> default it has units, but to strip them off you'll just need to >>>> >> cast to >>>> >> ndarray: >>>> >> >>>> >> array_without_units = array.v >>>> >> >>>> >> >>>> >>> >>>> >>> Also, is there a way to load multiple particle types? >>>> >>> >>>> >>> Do I just need to stack the particles into the array here? >>>> >>> >>>> >>> data = {'particle_position_x': pos[:,0], >>>> >>> 'particle_position_y': pos[:,1], >>>> >>> 'particle_position_z': pos[:,2], >>>> >>> 'particle_mass': np.array([mpart]*npart)} >>>> >>> >>>> >>> Then feed it in as usual? >>>> >> >>>> >> >>>> >> That's right, although if the particle masses are different for >>>> >> the >>>> >> different particle types that code snippet will need to be >>>> >> generalized to >>>> >> handle that. >>>> >> >>>> >> I think in principle it should be possible to make load_particles >>>> >> handle different particle types just like an SPH dataset that >>>> >> contains >>>> >> multiple particle types, but right now that hasn't been >>>> >> implemented yet. >>>> >> >>>> >>> >>>> >>> >>>> >>> Brendan >>>> >>> >>>> >>> >>>> >>> On Thu, Jun 5, 2014 at 9:44 PM, Nathan Goldbaum >>>> >>> <nathan12343@gmail.com> wrote: >>>> >>>> >>>> >>>> That's right, you can set that via the bbox keyword
wrote: than the like parameter
>>>> >>>> for >>>> >>>> load_particles. I'd urge you to take a look at the docstrings >>>> >>>> and source >>>> >>>> code for load_particles. >>>> >>>> >>>> >>>> >>>> >>>> On Thu, Jun 5, 2014 at 6:34 PM, Brendan Griffen >>>> >>>> <brendan.f.griffen@gmail.com> wrote: >>>> >>>>> >>>> >>>>> Thanks very much Nathan. I tried to load in my own data but I >>>> >>>>> think there are too many particles or I have to specifically >>>> >>>>> set the domain >>>> >>>>> size. >>>> >>>>> >>>> >>>>> In this area: >>>> >>>>> >>>> >>>>> data = {'particle_position_x': pos[:,0], >>>> >>>>> 'particle_position_y': pos[:,1], >>>> >>>>> 'particle_position_z': pos[:,2], >>>> >>>>> 'particle_mass': np.array([mpart]*npart)} >>>> >>>>> >>>> >>>>> ds = yt.load_particles(data, length_unit=3.08e24, >>>> >>>>> mass_unit=1.9891e36) >>>> >>>>> ad = ds.all_data() >>>> >>>>> print ad['deposit', 'all_cic'] >>>> >>>>> >>>> >>>>> In [3]: run ytcic.py >>>> >>>>> yt : [INFO ] 2014-06-05 21:29:06,183 Parameters: >>>> >>>>> current_time >>>> >>>>> = 0.0 >>>> >>>>> yt : [INFO ] 2014-06-05 21:29:06,183 Parameters: >>>> >>>>> domain_dimensions = [2 2 2] >>>> >>>>> yt : [INFO ] 2014-06-05 21:29:06,184 Parameters: >>>> >>>>> domain_left_edge = [ 0. 0. 0.] >>>> >>>>> yt : [INFO ] 2014-06-05 21:29:06,185 Parameters: >>>> >>>>> domain_right_edge = [ 1. 1. 1.] >>>> >>>>> yt : [INFO ] 2014-06-05 21:29:06,185 Parameters: >>>> >>>>> cosmological_simulation = 0.0 >>>> >>>>> yt : [INFO ] 2014-06-05 21:29:06,188 Allocating for >>>> >>>>> 1.342e+08 >>>> >>>>> particles >>>> >>>>> >>>> >>>>> >>>> >>>>>
>>>> >>>>> YTDomainOverflow Traceback (most >>>> >>>>> recent >>>> >>>>> call last) >>>> >>>>> >>>> >>>>> >>>> >>>>> /nfs/blank/h4231/bgriffen/data/lib/yt-x86_64/lib/python2.7/site-packages/IPython/utils/py3compat.pyc >>>> >>>>> in execfile(fname, *where) >>>> >>>>> 202 else: >>>> >>>>> 203 filename = fname >>>> >>>>> --> 204 __builtin__.execfile(filename, *where) >>>> >>>>> >>>> >>>>> >>>> >>>>> >>>> >>>>> /nfs/blank/h4231/bgriffen/work/projects/caterpillar/c2ray/cic/ytcic.py in >>>> >>>>> <module>() >>>> >>>>> 52 >>>> >>>>> 53 ad = ds.all_data() >>>> >>>>> ---> 54 print ad['deposit', 'all_cic'] >>>> >>>>> 55 slc = yt.SlicePlot(ds, 2, ('deposit', 'all_cic')) >>>> >>>>> 56 slc.set_figure_size(4) >>>> >>>>> >>>> >>>>> >>>> >>>>> >>>> >>>>> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc >>>> >>>>> in __getitem__(self, key) >>>> >>>>> 205 Returns a single field. Will add if necessary. >>>> >>>>> 206 """ >>>> >>>>> --> 207 f = self._determine_fields([key])[0] >>>> >>>>> 208 if f not in self.field_data and key not in >>>> >>>>> self.field_data: >>>> >>>>> 209 if f in self._container_fields: >>>> >>>>> >>>> >>>>> >>>> >>>>> >>>> >>>>> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc >>>> >>>>> in _determine_fields(self, fields) >>>> >>>>> 453 raise YTFieldNotParseable(field) >>>> >>>>> 454 ftype, fname = field >>>> >>>>> --> 455 finfo = self.pf._get_field_info(ftype, >>>> >>>>> fname) >>>> >>>>> 456 else: >>>> >>>>> 457 fname = field >>>> >>>>> >>>> >>>>> >>>> >>>>> >>>> >>>>> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/static_output.pyc >>>> >>>>> in _get_field_info(self, ftype, fname) >>>> >>>>> 445 _last_finfo = None >>>> >>>>> 446 def _get_field_info(self, ftype, fname = None): >>>> >>>>> --> 447 self.index >>>> >>>>> 448 if fname is None: >>>> >>>>> 449 ftype, fname = "unknown", ftype >>>> >>>>> >>>> >>>>> >>>> >>>>> >>>> >>>>> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/static_output.pyc >>>> >>>>> in index(self) >>>> >>>>> 277 raise RuntimeError("You should not >>>> >>>>> instantiate Dataset.") >>>> >>>>> 278 self._instantiated_index = >>>> >>>>> self._index_class( >>>> >>>>> --> 279 self, dataset_type=self.dataset_type) >>>> >>>>> 280 # Now we do things that we need an >>>> >>>>> instantiated index for >>>> >>>>> 281 # ...first off, we create our field_info >>>> >>>>> now. >>>> >>>>> >>>> >>>>> >>>> >>>>> >>>> >>>>> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/frontends/stream/data_structures.pyc >>>> >>>>> in __init__(self, pf, dataset_type) >>>> >>>>> 942 def __init__(self, pf, dataset_type = None): >>>> >>>>> 943 self.stream_handler = pf.stream_handler >>>> >>>>> --> 944 super(StreamParticleIndex, self).__init__(pf, >>>> >>>>> dataset_type) >>>> >>>>> 945 >>>> >>>>> 946 def _setup_data_io(self): >>>> >>>>> >>>> >>>>> >>>> >>>>> >>>> >>>>> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/geometry/particle_geometry_handler.pyc >>>> >>>>> in __init__(self, pf, dataset_type) >>>> >>>>> 48 self.directory = >>>> >>>>> os.path.dirname(self.index_filename) >>>> >>>>> 49 self.float_type = np.float64 >>>> >>>>> ---> 50 super(ParticleIndex, self).__init__(pf, >>>> >>>>> dataset_type) >>>> >>>>> 51 >>>> >>>>> 52 def _setup_geometry(self): >>>> >>>>> >>>> >>>>> >>>> >>>>> >>>> >>>>> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/geometry/geometry_handler.pyc >>>> >>>>> in __init__(self, pf, dataset_type) >>>> >>>>> 54 >>>> >>>>> 55 mylog.debug("Setting up domain geometry.") >>>> >>>>> ---> 56 self._setup_geometry() >>>> >>>>> 57 >>>> >>>>> 58 mylog.debug("Initializing data grid data IO") >>>> >>>>> >>>> >>>>> >>>> >>>>> >>>> >>>>> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/geometry/particle_geometry_handler.pyc >>>> >>>>> in _setup_geometry(self) >>>> >>>>> 52 def _setup_geometry(self): >>>> >>>>> 53 mylog.debug("Initializing Particle Geometry >>>> >>>>> Handler.") >>>> >>>>> ---> 54 self._initialize_particle_handler() >>>> >>>>> 55 >>>> >>>>> 56 >>>> >>>>> >>>> >>>>> >>>> >>>>> >>>> >>>>> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/geometry/particle_geometry_handler.pyc >>>> >>>>> in _initialize_particle_handler(self) >>>> >>>>> 87 pf.domain_left_edge, >>>> >>>>> pf.domain_right_edge, >>>> >>>>> 88 [N, N, N], len(self.data_files)) >>>> >>>>> ---> 89 self._initialize_indices() >>>> >>>>> 90 self.oct_handler.finalize() >>>> >>>>> 91 self.max_level = self.oct_handler.max_level >>>> >>>>> >>>> >>>>> >>>> >>>>> >>>> >>>>> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/geometry/particle_geometry_handler.pyc >>>> >>>>> in _initialize_indices(self) >>>> >>>>> 109 npart = >>>> >>>>> sum(data_file.total_particles.values()) >>>> >>>>> 110 morton[ind:ind + npart] = \ >>>> >>>>> --> 111 self.io._initialize_index(data_file, >>>> >>>>> self.regions) >>>> >>>>> 112 ind += npart >>>> >>>>> 113 morton.sort() >>>> >>>>> >>>> >>>>> >>>> >>>>> >>>> >>>>> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/frontends/stream/io.pyc in >>>> >>>>> _initialize_index(self, data_file, regions) >>>> >>>>> 144 raise YTDomainOverflow(pos.min(axis=0), >>>> >>>>> pos.max(axis=0), >>>> >>>>> 145 >>>> >>>>> data_file.pf.domain_left_edge, >>>> >>>>> --> 146 >>>> >>>>> data_file.pf.domain_right_edge) >>>> >>>>> 147 regions.add_data_file(pos, >>>> >>>>> data_file.file_id) >>>> >>>>> 148 morton.append(compute_morton( >>>> >>>>> >>>> >>>>> YTDomainOverflow: Particle bounds [ 0. 0. 0.] and [ >>>> >>>>> 99.99999237 >>>> >>>>> 99.99999237 99.99999237] exceed domain bounds [ 0. 0. 0.] >>>> >>>>> code_length and >>>> >>>>> [ 1. 1. 1.] code_length >>>> >>>>> >>>> >>>>> >>>> >>>>> >>>> >>>>> On Thu, Jun 5, 2014 at 8:22 PM, Nathan Goldbaum >>>> >>>>> <nathan12343@gmail.com> wrote: >>>> >>>>>> >>>> >>>>>> Here's a worked out example that does what you're looking for >>>> >>>>>> using a fake 1 million particle dataset: >>>> >>>>>> >>>> >>>>>> >>>> >>>>>> http://nbviewer.ipython.org/gist/ngoldbaum/546d37869aafe71cfe38 >>>> >>>>>> >>>> >>>>>> In this notebook I make use of two key yt features: >>>> >>>>>> `load_particles`, and `covering_grid`. >>>> >>>>>> >>>> >>>>>> load_particles creates a "stream" dataset based on in-memory >>>> >>>>>> data >>>> >>>>>> fed in as a numpy array. This dataset acts just like an >>>> >>>>>> on-disk simulation >>>> >>>>>> dataset, but doesn't come with the baggage of needing to write >>>> >>>>>> a custom >>>> >>>>>> frontend to read a specific data format off disk. >>>> >>>>>> >>>> >>>>>> covering_grid is a way to generate uniform resolution data >>>> >>>>>> from >>>> >>>>>> an AMR dataset. It acts like a python dictionary where the >>>> >>>>>> keys are field >>>> >>>>>> names and returns 3D numpy arrays of whatever uniform >>>> >>>>>> resolution you specify >>>> >>>>>> when you create the covering_grid. >>>> >>>>>> >>>> >>>>>> Note that if you're using load_particles all of your data >>>> >>>>>> needs >>>> >>>>>> to live in memory. If your data is too big for that you'll >>>> >>>>>> need to write a >>>> >>>>>> frontend for your data format or use a memmap to an on-disk >>>> >>>>>> file somehow. >>>> >>>>>> I'm not an expert on that but others on the list should be >>>> >>>>>> able to help out. >>>> >>>>>> >>>> >>>>>> Hope that gets you well on your way :) >>>> >>>>>> >>>> >>>>>> -Nathan >>>> >>>>>> >>>> >>>>>> >>>> >>>>>> On Thu, Jun 5, 2014 at 5:04 PM, Desika Narayanan >>>> >>>>>> <dnarayan@haverford.edu> wrote: >>>> >>>>>>> >>>> >>>>>>> Hey Brendan, >>>> >>>>>>> >>>> >>>>>>> A couple of extra tools you might find helpful in conjunction >>>> >>>>>>> with Nathan's example of depositing the particles onto an >>>> >>>>>>> octree are at: >>>> >>>>>>> >>>> >>>>>>> http://paste.yt-project.org/show/4737/ >>>> >>>>>>> >>>> >>>>>>> Where I load a gadget snapshot, and then recover the >>>> >>>>>>> coordinates >>>> >>>>>>> and width of each cell. >>>> >>>>>>> >>>> >>>>>>> In response to your last question - the particles are >>>> >>>>>>> deposited >>>> >>>>>>> into an octree grid (so, you'll see that the cell sizes >>>> >>>>>>> aren't all the same >>>> >>>>>>> size). I don't know if depositing onto a regular NxNxN mesh >>>> >>>>>>> is possible, >>>> >>>>>>> though would be interested to hear if so. >>>> >>>>>>> >>>> >>>>>>> -d >>>> >>>>>>> >>>> >>>>>>> >>>> >>>>>>> On Thu, Jun 5, 2014 at 7:58 PM, Brendan Griffen >>>> >>>>>>> <brendan.f.griffen@gmail.com> wrote: >>>> >>>>>>>> >>>> >>>>>>>> Thanks. I'll get the "bleeding edge" version first then try >>>> >>>>>>>> your suggestions. Though I want to return the NxNxN array >>>> >>>>>>>> and be able to >>>> >>>>>>>> write this mesh to a file. It is *only* using the cic part >>>> >>>>>>>> of yt and it >>>> >>>>>>>> should return the mesh to be written? Just wanted to >>>> >>>>>>>> clarify? >>>> >>>>>>>> >>>> >>>>>>>> Thanks. >>>> >>>>>>>> Brendan >>>> >>>>>>>> >>>> >>>>>>>> >>>> >>>>>>>> On Thu, Jun 5, 2014 at 6:49 PM, Nathan Goldbaum >>>> >>>>>>>> <nathan12343@gmail.com> wrote: >>>> >>>>>>>>> >>>> >>>>>>>>> >>>> >>>>>>>>> >>>> >>>>>>>>> >>>> >>>>>>>>> On Thu, Jun 5, 2014 at 3:36 PM, John ZuHone >>>> >>>>>>>>> <jzuhone@gmail.com> wrote: >>>> >>>>>>>>>> >>>> >>>>>>>>>> Hi Brendan, >>>> >>>>>>>>>> >>>> >>>>>>>>>> Which version of yt are you using? >>>> >>>>>>>>>> >>>> >>>>>>>>>> If you're using 3.0, this is actually fairly easy. If you >>>> >>>>>>>>>> look in yt.fields.particle_fields.py, around line 85, you >>>> >>>>>>>>>> can see how this >>>> >>>>>>>>>> is done for the "particle_density" and "particle_mass" >>>> >>>>>>>>>> fields. Basically you >>>> >>>>>>>>>> can call a "deposit" method which takes the particle field >>>> >>>>>>>>>> quantity you want >>>> >>>>>>>>>> deposited and deposits it into cells. The underlying >>>> >>>>>>>>>> calculation is done >>>> >>>>>>>>>> using Cython, so it's fast. >>>> >>>>>>>>> >>>> >>>>>>>>> >>>> >>>>>>>>> And you shouldn't ever actually need to call these >>>> >>>>>>>>> "deposit" >>>> >>>>>>>>> functions, since "deposit" is exposed as a field type for >>>> >>>>>>>>> all datasets that >>>> >>>>>>>>> contain particles. >>>> >>>>>>>>> >>>> >>>>>>>>> Here is a notebook that does this for Enzo AMR data: >>>> >>>>>>>>> >>>> >>>>>>>>> >>>> >>>>>>>>> >>>> >>>>>>>>> http://nbviewer.ipython.org/gist/ngoldbaum/5e19e4e6cc2bf330149c >>>> >>>>>>>>> >>>> >>>>>>>>> This dataset contains about a million particles and >>>> >>>>>>>>> generates >>>> >>>>>>>>> a CIC deposition for the whole domain in about 6 seconds >>>> >>>>>>>>> from a cold start. >>>> >>>>>>>>> >>>> >>>>>>>>>> >>>> >>>>>>>>>> >>>> >>>>>>>>>> If you're using 2.x, then you can do the same thing, but >>>> >>>>>>>>>> it's >>>> >>>>>>>>>> not as straightforward. You can see how this works in >>>> >>>>>>>>>> yt.data_objects.universal_fields.py, around line 986, >>>> >>>>>>>>>> where the >>>> >>>>>>>>>> "particle_density" field is defined. Basically, it calls >>>> >>>>>>>>>> CICDeposit_3, which >>>> >>>>>>>>>> is in yt.utilities.lib.CICDeposit.pyx. >>>> >>>>>>>>>> >>>> >>>>>>>>>> Let me know if you need any more clarification. >>>> >>>>>>>>>> >>>> >>>>>>>>>> Best, >>>> >>>>>>>>>> >>>> >>>>>>>>>> John Z >>>> >>>>>>>>>> >>>> >>>>>>>>>> On Jun 5, 2014, at 6:07 PM, Brendan Griffen >>>> >>>>>>>>>> <brendan.f.griffen@gmail.com> wrote: >>>> >>>>>>>>>> >>>> >>>>>>>>>> > Hi, >>>> >>>>>>>>>> > >>>> >>>>>>>>>> > I was wondering if there were any Cython routines within >>>> >>>>>>>>>> > yt >>>> >>>>>>>>>> > which takes particle data and converts it into a >>>> >>>>>>>>>> > cloud-in-cell based mesh >>>> >>>>>>>>>> > which can be written to a file of my choosing. >>>> >>>>>>>>> >>>> >>>>>>>>> >>>> >>>>>>>>> What sort of mesh were you looking for? yt will internally >>>> >>>>>>>>> construct an octree if it is fed particle data. I'm not >>>> >>>>>>>>> sure whether this >>>> >>>>>>>>> octree can be saved to disk for later analysis. >>>> >>>>>>>>> >>>> >>>>>>>>> It's also possible to create a uniform resolution covering >>>> >>>>>>>>> grid containing field data for a deposited quantity, which >>>> >>>>>>>>> can be quite >>>> >>>>>>>>> easily saved to disk in a number of ways. >>>> >>>>>>>>> >>>> >>>>>>>>>> >>>> >>>>>>>>>> I heard a while ago there was some such functionality but >>>> >>>>>>>>>> it >>>> >>>>>>>>>> could be too far down the yt rabbit hole to be used as a >>>> >>>>>>>>>> standalone? Is this >>>> >>>>>>>>>> true? I have my own Python code for doing it but it just >>>> >>>>>>>>>> isn't fast enough >>>> >>>>>>>>>> and thought I'd ask the yt community if there were any >>>> >>>>>>>>>> wrapper tools >>>> >>>>>>>>>> available to boost the speed. >>>> >>>>>>>>>> > >>>> >>>>>>>>>> > Thanks. >>>> >>>>>>>>>> > Brendan >>>> >>>>>>>>>> > _______________________________________________ >>>> >>>>>>>>>> > yt-users mailing list >>>> >>>>>>>>>> > yt-users@lists.spacepope.org >>>> >>>>>>>>>> > >>>> >>>>>>>>>> > >>>> >>>>>>>>>> > http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >>>> >>>>>>>>>> >>>> >>>>>>>>>> _______________________________________________ >>>> >>>>>>>>>> yt-users mailing list >>>> >>>>>>>>>> yt-users@lists.spacepope.org >>>> >>>>>>>>>> >>>> >>>>>>>>>> >>>> >>>>>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >>>> >>>>>>>>> >>>> >>>>>>>>> >>>> >>>>>>>>> >>>> >>>>>>>>> _______________________________________________ >>>> >>>>>>>>> yt-users mailing list >>>> >>>>>>>>> yt-users@lists.spacepope.org >>>> >>>>>>>>> >>>> >>>>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >>>> >>>>>>>>> >>>> >>>>>>>> >>>> >>>>>>>> >>>> >>>>>>>> _______________________________________________ >>>> >>>>>>>> yt-users mailing list >>>> >>>>>>>> yt-users@lists.spacepope.org >>>> >>>>>>>> >>>> >>>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >>>> >>>>>>>> >>>> >>>>>>> >>>> >>>>>>> >>>> >>>>>>> _______________________________________________ >>>> >>>>>>> yt-users mailing list >>>> >>>>>>> yt-users@lists.spacepope.org >>>> >>>>>>> >>>> >>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >>>> >>>>>>> >>>> >>>>>> >>>> >>>>>> >>>> >>>>>> _______________________________________________ >>>> >>>>>> yt-users mailing list >>>> >>>>>> yt-users@lists.spacepope.org >>>> >>>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >>>> >>>>>> >>>> >>>>> >>>> >>>>> >>>> >>>>> _______________________________________________ >>>> >>>>> yt-users mailing list >>>> >>>>> yt-users@lists.spacepope.org >>>> >>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >>>> >>>>> >>>> >>>> >>>> >>>> >>>> >>>> _______________________________________________ >>>> >>>> yt-users mailing list >>>> >>>> yt-users@lists.spacepope.org >>>> >>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >>>> >>>> >>>> >>> >>>> >>> >>>> >>> _______________________________________________ >>>> >>> yt-users mailing list >>>> >>> yt-users@lists.spacepope.org >>>> >>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >>>> >>> >>>> >> >>>> >> >>>> >> _______________________________________________ >>>> >> yt-users mailing list >>>> >> yt-users@lists.spacepope.org >>>> >> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >>>> >> >>>> > >>>> > >>>> > _______________________________________________ >>>> > yt-users mailing list >>>> > yt-users@lists.spacepope.org >>>> > http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >>>> > >>>> >>>> >>>> _______________________________________________ >>>> yt-users mailing list >>>> yt-users@lists.spacepope.org >>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >>>> >>> >> >> >> _______________________________________________ >> yt-users mailing list >> yt-users@lists.spacepope.org >> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >> > > > _______________________________________________ > yt-users mailing list > yt-users@lists.spacepope.org > http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org > _______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
Hi Nathan, I'm not sure if you missed it but earlier Matt asked if there was some copying which shouldn't be happening? Brendan On Mon, Jun 9, 2014 at 1:22 PM, Nathan Goldbaum <nathan12343@gmail.com> wrote:
On Mon, Jun 9, 2014 at 12:32 AM, Brendan Griffen < brendan.f.griffen@gmail.com> wrote:
Thanks Nathan. Comparing the two though:
26 1536.664 MiB 1130.480 MiB uniform_array = grid_object['deposit', 'all_cic']
24 1285.703 MiB 890.539 MiB cic_density = ad["deposit", "all_cic"]
Isn't it the uniform grid which uses the most memory? I'm running the profiling now and will get back to you. I hope in spite of the fact that it does crash it will still give me some useful output.
That's true - I was thinking it might be more memory efficient since there is no need to construct the covering grid in addition to the octree.
Brendan
On Mon, Jun 9, 2014 at 2:05 AM, Nathan Goldbaum <nathan12343@gmail.com> wrote:
Hey Brendan,
Could you try running your script using the memory_profiler module on pypi?
Here's an example script that uses the memory profiler: http://paste.yt-project.org/show/4748/
and the output for that script: http://paste.yt-project.org/show/4749/
For what it's worth, it does indeed look like matt's suggestion to use load_uniform_grid is an option, and might be more memory efficient in the end since you will go directly to the uniform grid you want without creating an octree. Here's an example: http://paste.yt-project.org/show/4750/
Here's the memory usage information for that example: http://paste.yt-project.org/show/4751/
I used a 256^3 uniform grid with normally distributed random data - I'm not sure whether it will also be more memory efficient in your case.
On Sun, Jun 8, 2014 at 9:32 PM, Brendan Griffen < brendan.f.griffen@gmail.com> wrote:
This is the full error if it helps at all? It is indeed, loading in all of the quantities.
Loading particles... --> Loading particle type: 1 yt : [INFO ] 2014-06-08 15:12:05,540 Parameters: current_time = 0.0 yt : [INFO ] 2014-06-08 15:12:05,540 Parameters: domain_dimensions = [2 2 2] yt : [INFO ] 2014-06-08 15:12:05,541 Parameters: domain_left_edge = [ 0. 0. 0.] yt : [INFO ] 2014-06-08 15:12:05,542 Parameters: domain_right_edge = [ 100. 100. 100.] yt : [INFO ] 2014-06-08 15:12:05,542 Parameters: cosmological_simulation = 0.0 yt : [INFO ] 2014-06-08 15:12:05,548 Allocating for 1.074e+09 particles yt : [INFO ] 2014-06-08 15:16:16,195 Identified 7.584e+07 octs yt : [INFO ] 2014-06-08 15:16:16,299 Loading field plugins. yt : [INFO ] 2014-06-08 15:16:16,299 Loaded angular_momentum (8 new fields) yt : [INFO ] 2014-06-08 15:16:16,299 Loaded astro (14 new fields) yt : [INFO ] 2014-06-08 15:16:16,300 Loaded cosmology (20 new fields) yt : [INFO ] 2014-06-08 15:16:16,300 Loaded fluid (56 new fields) yt : [INFO ] 2014-06-08 15:16:16,301 Loaded fluid_vector (88 new fields) yt : [INFO ] 2014-06-08 15:16:16,301 Loaded geometric (103 new fields) yt : [INFO ] 2014-06-08 15:16:16,301 Loaded local (103 new fields) yt : [INFO ] 2014-06-08 15:16:16,302 Loaded magnetic_field (109 new fields) yt : [INFO ] 2014-06-08 15:16:16,302 Loaded species (109 new fields)
--------------------------------------------------------------------------- MemoryError Traceback (most recent call last) /nfs/blank/h4231/bgriffen/data/lib/yt-x86_64/lib/python2.7/site-packages/IPython/utils/py3compat.pyc in execfile(fname, *where) 202 else: 203 filename = fname --> 204 __builtin__.execfile(filename, *where)
/nfs/blank/h4231/bgriffen/work/projects/caterpillar/c2ray/cic/ytcic.py in <module>() 99 slc.set_figure_size(4) 100 slc.save() --> 101 102 for ndim in dimlist: 103 print
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc in __getitem__(self, key) 218 return self.field_data[f] 219 else: --> 220 self.get_data(f) 221 # fi.units is the unit expression string. We depend on the registry 222 # hanging off the dataset to define this unit object.
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc in get_data(self, fields) 627 628 fields_to_generate += gen_fluids + gen_particles --> 629 self._generate_fields(fields_to_generate) 630 631 def _generate_fields(self, fields_to_generate):
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc in _generate_fields(self, fields_to_generate) 644 fi = self.pf._get_field_info(*field) 645 try: --> 646 fd = self._generate_field(field) 647 if type(fd) == np.ndarray: 648 fd = self.pf.arr(fd, fi.units)
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc in _generate_field(self, field) 255 tr = self._generate_particle_field(field) 256 else: --> 257 tr = self._generate_fluid_field(field) 258 if tr is None: 259 raise YTCouldNotGenerateField(field, self.pf)
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc in _generate_fluid_field(self, field) 273 finfo.check_available(gen_obj) 274 except NeedsGridType as ngt_exception: --> 275 rv = self._generate_spatial_fluid(field, ngt_exception.ghost_zones) 276 else: 277 rv = finfo(gen_obj)
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc in _generate_spatial_fluid(self, field, ngz) 289 o = self._current_chunk.objs[0] 290 with o._activate_cache(): --> 291 ind += o.select(self.selector, self[field], rv, ind) 292 else: 293 chunks = self.index._chunk(self, "spatial", ngz = ngz)
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc in __getitem__(self, key) 218 return self.field_data[f] 219 else: --> 220 self.get_data(f) 221 # fi.units is the unit expression string. We depend on the registry 222 # hanging off the dataset to define this unit object.
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc in get_data(self, fields) 627 628 fields_to_generate += gen_fluids + gen_particles --> 629 self._generate_fields(fields_to_generate) 630 631 def _generate_fields(self, fields_to_generate):
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc in _generate_fields(self, fields_to_generate) 644 fi = self.pf._get_field_info(*field) 645 try: --> 646 fd = self._generate_field(field) 647 if type(fd) == np.ndarray: 648 fd = self.pf.arr(fd, fi.units)
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc in _generate_field(self, field) 255 tr = self._generate_particle_field(field) 256 else: --> 257 tr = self._generate_fluid_field(field) 258 if tr is None: 259 raise YTCouldNotGenerateField(field, self.pf)
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc in _generate_fluid_field(self, field) 275 rv = self._generate_spatial_fluid(field, ngt_exception.ghost_zones) 276 else: --> 277 rv = finfo(gen_obj) 278 return rv 279
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/fields/derived_field.pyc in __call__(self, data) 176 "for %s" % (self.name,)) 177 with self.unit_registry(data): --> 178 dd = self._function(self, data) 179 for field_name in data.keys(): 180 if field_name not in original_fields:
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/fields/particle_fields.pyc in particle_cic(field, data) 113 def particle_cic(field, data): 114 pos = data[ptype, coord_name] --> 115 d = data.deposit(pos, [data[ptype, mass_name]], method = "cic") 116 d = data.apply_units(d, data[ptype, mass_name].units) 117 d /= data["index", "cell_volume"]
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/octree_subset.pyc in deposit(self, positions, fields, method) 167 mylog.debug("Depositing %s (%s^3) particles into %s Octs", 168 positions.shape[0], positions.shape[0]**0.3333333, nvals[-1]) --> 169 pos = np.asarray(positions.convert_to_units("code_length"), 170 dtype="float64") 171 # We should not need the following if we know in advance all our fields
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/units/yt_array.pyc in convert_to_units(self, units) 366 367 self.units = new_units --> 368 self *= conversion_factor 369 return self 370
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/units/yt_array.pyc in __imul__(self, other) 667 """ See __mul__. """ 668 oth = sanitize_units_mul(self, other) --> 669 return np.multiply(self, oth, out=self) 670 671 def __div__(self, right_object):
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/units/yt_array.pyc in __array_wrap__(self, out_arr, context) 966 # casting to YTArray avoids creating a YTQuantity with size > 1 967 return YTArray(np.array(out_arr, unit)) --> 968 return ret_class(np.array(out_arr), unit) 969 970
MemoryError:
On Sun, Jun 8, 2014 at 10:50 PM, Matthew Turk <matthewturk@gmail.com> wrote:
Hi Brendan,
Hi Matt,
Thanks for your detailed email. Forgive my naivety but why do you need the oct-tree in the first place? I have a my own fortran code for constructing a cloud in cell mesh and it uses very little overhead (just the n^3 grid and the particle data itself). I then calculate the dx,dy,dzs to the nearest 8 grid points and distribute accordingly in a omp loop which is done in a fraction of a second. Does the situation with yt come about (oct
On Sun, Jun 8, 2014 at 9:21 PM, Brendan Griffen <brendan.f.griffen@gmail.com> wrote: tree etc.)
necessarily because of the way it handles particle data? Is it essentially used to map the particles to domains in the grid or something?
That's not naive at all. There are two reasons --
1) The octree is used for indexing for neighbor lookups and early-termination of region selection for particles 2) The octree is used to estimate the "required resolution" for any operation that requires a space-filling value. (i.e., any time that a particle becomes a volume.)
Projections in yt are adaptive, in that they project down to the finest appropriate resolution. There's also the "arbitrary_grid" operation, which does precisely what you're describing, but as it stands right now the octree gets constructed at time of instantiation of the indexing system. Thinking it over, you may be able to avoid that completely by not using load_particles and instead using load_uniform_grid and supplying your desired dimensions. The field names should be the same.
The machine I max memory on has 128GB and the snapshots are using
1024^3
particles. Do you have any idea of how much memory the oct-tree uses as a function of particle/grid number? I am going to try on a 256GB machine (though this is a bit of a hassle). I'll see how I go.
I am disappointed that it's blowing out your RAM. This week I will try to get some memory profiling done. Could you file a bug to this effect, which will help me track it? Peak memory usage during indexing should only be 64 bits * Nparticles, unless you're using load_particles, in which case all the fields will *also* have to be in memory. It's about 8 gigabytes per field. So, I think there's something going wrong.
Thanks.
Brendan
On Sun, Jun 8, 2014 at 6:25 PM, Matthew Turk <matthewturk@gmail.com>
> > Hi all, > > I feel like I owe a brief explanation of why things are tricky right > now, what we're planning on doing, and how we're experimenting and > developing. > > Presently, the particle geometry handlers build a single mesh from all > particles in the dataset, along with a coarse bitmask that correlates > files to regions in the domain. This requires the allocation of a > single int64 array of size Nparticles, which is sorted in place and > then fed into an octree construction algorithm that then spits back > out the mesh. Each octree component contains 3 64-bit integers and > eitehr a void pointer or a pointer to eight other octs. Increasing > n_ref decreases the number of octs in this mesh; when smoothing > operaitons are conducted, a second "index" mesh is created for looking > up particles near mesh points. Mesh points are used for adaptive > resolution smoothing and other "deposit particles on the grid somehow" > operations (including SPH kernel). > > Anyway, because right now it requires a global mesh to be constructed, > this is expensive and requires holding a 64-bit integer in memory for > each particle. I think if you're loading the particles in differently > there is some additional overhead as well, but I'm still a bit > surprised you OOM on a 1024^3 dataset. > > In general, we don't *need* this global mesh; is can be constructed as > required, which would speed up both the initial index phase as well as > the final meshing process. I got about 50% of the way to implementing > this last fall, but because of various concerns and deadlines I > haven't finished it. I intend to get back to it probably in July, > right after we put out a 3.0, so that we can have it in time for 3.1. > In principle this will make the particle codes much more similar to > ARTIO, in that the mesh will be constructed only as required and > discarded when no longer required, which will make them much more > memory efficient. > > But, getting a single mesh for extremely large data is a very high > priority; right now for the 10240^3 run we've been loading up > individual sub-chunks, which I want to stop doing. > > From the technical perspective, these are the things that need to > happen on the yt side for particle datasets to move to this "lazy" > mode of loading; most of this is based on things learned from 2HOT and > ARTIO, and will involve converting to a forest-of-octrees. > > * Split into spatially-organized subchunks of ParticleOctreeSubset > objects, such that these map 1:Nfiles, and that can be constructed on > the fly. > * Construct a dual-mesh of the bitmask "ParticleRegion" object that > will help with identifying neighbors to a given oct cell, so that if > we're inside one octree we know which neighbor octrees to grab if we > need particles for smoothing things (fast boundary particle > identification is later down the road) > * Parallel sort of particles, or using the parallel ring function; > may not be necessary after all > > All of this is doable, and I'd be happy to work with people if
> like to take a shot at implementing it, but I've mostly put it on my > list for post-3.0. > > -Matt > > On Sun, Jun 8, 2014 at 2:43 PM, Nathan Goldbaum < nathan12343@gmail.com> > wrote: > > > > > > > > On Sun, Jun 8, 2014 at 12:27 PM, Brendan Griffen > > <brendan.f.griffen@gmail.com> wrote: > >> > >> Also, how do I construct just a zero filled yt array with dimensions > >> (ndim,ndim,ndim)? Thanks > > > > > > > > from yt import YTArray > > from numpy import np > > > > arr = YTArray(np.zeros([ndim, ndim, ndim]), input_units=units_string) > > > > or alternatively: > > > > from yt.units import kiloparsec > > > > arr = kiloparsec*np.zeros([ndim, ndim, ndim]) > > > > it doesn't have to be kiloparsec - you can compose the units you want > > out of > > any of the unit symbols that live in yt.units. > > > > See this page for a ton more detail about yt's new unit system: > > http://yt-project.org/docs/dev-3.0/analyzing/units/index.html > > > >> > >> > >> Brendan > >> > >> > >> On Sun, Jun 8, 2014 at 3:26 PM, Brendan Griffen > >> <brendan.f.griffen@gmail.com> wrote: > >>> > >>> Hi, > >>> > >>> Since I get memory errors. Could I not just read in the blocks of the > >>> output individually then basically stack the mesh each time. That way > >>> not > >>> every single particle of the snapshot has to be loaded at the same > >>> time. > >>> Would that just be a case of doing > >>> > >>> level = int(math.log(ndim,2)) > >>> cg = ds.covering_grid(level=level, > >>> left_edge=[0,0,0],dims=[ndim,ndim,ndim]) > >>> arr = cg['deposit', 'all_density'] > >>> arrall += arr > >>> > >>> in a loop over each HDF5 block? > > > > > > It's likely that the memory use is dominated by the octree rather
> > the > > covering grid. This is with 1024^3 particles, correct? > > > > You can probably significantly reduce the memory used by the octree by > > increasing n_ref in the call to load_particles. > > > > See this page for more detail about load_particles: > > > > http://yt-project.org/docs/dev-3.0/examining/loading_data.html#generic-parti... > > > > Larger n_ref means fewer octree cells (lower resolution), but it also > > means > > lower poisson noise and lower memory use. > > > > Alternatively, as Matt suggested, you could break your 1024^3 ensemble > > of > > particles up into chunks, loop over the chunk, creating a particle > > octree > > and then a covering grid for each subset of the particles. Your final > > covering grid is just the sub of the covering grids for each subset of > > particles. > > > >>> > >>> > >>> Thanks. > >>> Brendan > >>> > >>> > >>> > >>> > >>> On Fri, Jun 6, 2014 at 7:26 PM, Matthew Turk < matthewturk@gmail.com> > >>> wrote: > >>>> > >>>> > >>>> On Jun 6, 2014 4:54 PM, "Brendan Griffen" > >>>> <brendan.f.griffen@gmail.com> > >>>> wrote: > >>>> > > >>>> > OK great. It is very low resolution but it worked. Thanks for all > >>>> > your > >>>> > help. My higher resolution run 1024^3 in 100 Mpc seems to crash on > >>>> > 128GB > >>>> > memory machine. I might have to look elsewhere. > >>>> > > >>>> > >>>> If you are looking for low resolution extraction you can tune
> >>>> memory usage by changing the parameter n_ref to something higher. > >>>> > >>>> Supporting extremely large datasets in a single mesh is on the > >>>> roadmap > >>>> for the late summer or fall, after a 3.0 release goes out. For now > >>>> you can > >>>> also extract before you load in; this is sort of how we are > >>>> supporting an > >>>> INCITE project with very large particle counts. > >>>> > >>>> > >>>> > Also, I normally use Canopy distribution but I just use an alias to > >>>> > loadyt which erases my PYTHONPATH and I can't access scipy and a > >>>> > few other > >>>> > libraries any more. What is the best practice here? Should I just > >>>> > manually > >>>> > export PYTHONPATH and point to the libraries need in canopy or can > >>>> > they play > >>>> > nice together? > >>>> > > >>>> > Thanks. > >>>> > > >>>> > BG > >>>> > > >>>> > > >>>> > On Fri, Jun 6, 2014 at 2:54 PM, Nathan Goldbaum > >>>> > <nathan12343@gmail.com> wrote: > >>>> >> > >>>> >> > >>>> >> > >>>> >> > >>>> >> On Fri, Jun 6, 2014 at 11:48 AM, Brendan Griffen > >>>> >> <brendan.f.griffen@gmail.com> wrote: > >>>> >>> > >>>> >>> OK great. Thanks. I just wanted a homogeneous mesh. 512^3 with no > >>>> >>> nesting of any kind. Though when I plot the image it looks
> >>>> >>> it is > >>>> >>> assigning particles incorrectly (low resolution on the outside). > >>>> >>> This is > >>>> >>> just a test image. > >>>> >>> > >>>> >> > >>>> >> The SlicePlot is visualizing the octree so there is less > >>>> >> resolution > >>>> >> where there are fewer particles. If you want to visualize
> >>>> >> covering grid > >>>> >> you're going to need to visualize that separately. > >>>> >> > >>>> >>> > >>>> >>> ds = yt.load_particles(data, length_unit=3.08e24, > >>>> >>> mass_unit=1.9891e33,bbox=bbox) > >>>> >>> > >>>> >>> ad = ds.all_data() > >>>> >>> print ad['deposit', 'all_cic'] > >>>> >>> slc = yt.SlicePlot(ds, 2, ('deposit', 'all_cic')) > >>>> >>> slc.set_figure_size(4) > >>>> >>> cg = ds.covering_grid(level=9, > >>>> >>> left_edge=[0,0,0],dims=[512,512,512]) > >>>> >>> > >>>> >> > >>>> >> To actually produce the uniform resolution ndarray, you're going > >>>> >> to > >>>> >> need to do something like: > >>>> >> > >>>> >> array = cg[('deposit', 'all_cic')] > >>>> >> > >>>> >> array will then be a 3D array you can do whatever you want with. > >>>> >> By > >>>> >> default it has units, but to strip them off you'll just need to > >>>> >> cast to > >>>> >> ndarray: > >>>> >> > >>>> >> array_without_units = array.v > >>>> >> > >>>> >> > >>>> >>> > >>>> >>> Also, is there a way to load multiple particle types? > >>>> >>> > >>>> >>> Do I just need to stack the particles into the array here? > >>>> >>> > >>>> >>> data = {'particle_position_x': pos[:,0], > >>>> >>> 'particle_position_y': pos[:,1], > >>>> >>> 'particle_position_z': pos[:,2], > >>>> >>> 'particle_mass': np.array([mpart]*npart)} > >>>> >>> > >>>> >>> Then feed it in as usual? > >>>> >> > >>>> >> > >>>> >> That's right, although if the particle masses are different for > >>>> >> the > >>>> >> different particle types that code snippet will need to be > >>>> >> generalized to > >>>> >> handle that. > >>>> >> > >>>> >> I think in principle it should be possible to make load_particles > >>>> >> handle different particle types just like an SPH dataset
> >>>> >> contains > >>>> >> multiple particle types, but right now that hasn't been > >>>> >> implemented yet. > >>>> >> > >>>> >>> > >>>> >>> > >>>> >>> Brendan > >>>> >>> > >>>> >>> > >>>> >>> On Thu, Jun 5, 2014 at 9:44 PM, Nathan Goldbaum > >>>> >>> <nathan12343@gmail.com> wrote: > >>>> >>>> > >>>> >>>> That's right, you can set that via the bbox keyword
wrote: they'd than the like the that parameter
> >>>> >>>> for > >>>> >>>> load_particles. I'd urge you to take a look at the docstrings > >>>> >>>> and source > >>>> >>>> code for load_particles. > >>>> >>>> > >>>> >>>> > >>>> >>>> On Thu, Jun 5, 2014 at 6:34 PM, Brendan Griffen > >>>> >>>> <brendan.f.griffen@gmail.com> wrote: > >>>> >>>>> > >>>> >>>>> Thanks very much Nathan. I tried to load in my own data but I > >>>> >>>>> think there are too many particles or I have to specifically > >>>> >>>>> set the domain > >>>> >>>>> size. > >>>> >>>>> > >>>> >>>>> In this area: > >>>> >>>>> > >>>> >>>>> data = {'particle_position_x': pos[:,0], > >>>> >>>>> 'particle_position_y': pos[:,1], > >>>> >>>>> 'particle_position_z': pos[:,2], > >>>> >>>>> 'particle_mass': np.array([mpart]*npart)} > >>>> >>>>> > >>>> >>>>> ds = yt.load_particles(data, length_unit=3.08e24, > >>>> >>>>> mass_unit=1.9891e36) > >>>> >>>>> ad = ds.all_data() > >>>> >>>>> print ad['deposit', 'all_cic'] > >>>> >>>>> > >>>> >>>>> In [3]: run ytcic.py > >>>> >>>>> yt : [INFO ] 2014-06-05 21:29:06,183 Parameters: > >>>> >>>>> current_time > >>>> >>>>> = 0.0 > >>>> >>>>> yt : [INFO ] 2014-06-05 21:29:06,183 Parameters: > >>>> >>>>> domain_dimensions = [2 2 2] > >>>> >>>>> yt : [INFO ] 2014-06-05 21:29:06,184 Parameters: > >>>> >>>>> domain_left_edge = [ 0. 0. 0.] > >>>> >>>>> yt : [INFO ] 2014-06-05 21:29:06,185 Parameters: > >>>> >>>>> domain_right_edge = [ 1. 1. 1.] > >>>> >>>>> yt : [INFO ] 2014-06-05 21:29:06,185 Parameters: > >>>> >>>>> cosmological_simulation = 0.0 > >>>> >>>>> yt : [INFO ] 2014-06-05 21:29:06,188 Allocating for > >>>> >>>>> 1.342e+08 > >>>> >>>>> particles > >>>> >>>>> > >>>> >>>>> > >>>> >>>>>
> >>>> >>>>> YTDomainOverflow Traceback (most > >>>> >>>>> recent > >>>> >>>>> call last) > >>>> >>>>> > >>>> >>>>> > >>>> >>>>> /nfs/blank/h4231/bgriffen/data/lib/yt-x86_64/lib/python2.7/site-packages/IPython/utils/py3compat.pyc > >>>> >>>>> in execfile(fname, *where) > >>>> >>>>> 202 else: > >>>> >>>>> 203 filename = fname > >>>> >>>>> --> 204 __builtin__.execfile(filename, *where) > >>>> >>>>> > >>>> >>>>> > >>>> >>>>> > >>>> >>>>> /nfs/blank/h4231/bgriffen/work/projects/caterpillar/c2ray/cic/ytcic.py in > >>>> >>>>> <module>() > >>>> >>>>> 52 > >>>> >>>>> 53 ad = ds.all_data() > >>>> >>>>> ---> 54 print ad['deposit', 'all_cic'] > >>>> >>>>> 55 slc = yt.SlicePlot(ds, 2, ('deposit', 'all_cic')) > >>>> >>>>> 56 slc.set_figure_size(4) > >>>> >>>>> > >>>> >>>>> > >>>> >>>>> > >>>> >>>>> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc > >>>> >>>>> in __getitem__(self, key) > >>>> >>>>> 205 Returns a single field. Will add if necessary. > >>>> >>>>> 206 """ > >>>> >>>>> --> 207 f = self._determine_fields([key])[0] > >>>> >>>>> 208 if f not in self.field_data and key not in > >>>> >>>>> self.field_data: > >>>> >>>>> 209 if f in self._container_fields: > >>>> >>>>> > >>>> >>>>> > >>>> >>>>> > >>>> >>>>> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc > >>>> >>>>> in _determine_fields(self, fields) > >>>> >>>>> 453 raise YTFieldNotParseable(field) > >>>> >>>>> 454 ftype, fname = field > >>>> >>>>> --> 455 finfo = self.pf._get_field_info(ftype, > >>>> >>>>> fname) > >>>> >>>>> 456 else: > >>>> >>>>> 457 fname = field > >>>> >>>>> > >>>> >>>>> > >>>> >>>>> > >>>> >>>>> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/static_output.pyc > >>>> >>>>> in _get_field_info(self, ftype, fname) > >>>> >>>>> 445 _last_finfo = None > >>>> >>>>> 446 def _get_field_info(self, ftype, fname = None): > >>>> >>>>> --> 447 self.index > >>>> >>>>> 448 if fname is None: > >>>> >>>>> 449 ftype, fname = "unknown", ftype > >>>> >>>>> > >>>> >>>>> > >>>> >>>>> > >>>> >>>>> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/static_output.pyc > >>>> >>>>> in index(self) > >>>> >>>>> 277 raise RuntimeError("You should not > >>>> >>>>> instantiate Dataset.") > >>>> >>>>> 278 self._instantiated_index = > >>>> >>>>> self._index_class( > >>>> >>>>> --> 279 self, dataset_type=self.dataset_type) > >>>> >>>>> 280 # Now we do things that we need an > >>>> >>>>> instantiated index for > >>>> >>>>> 281 # ...first off, we create our field_info > >>>> >>>>> now. > >>>> >>>>> > >>>> >>>>> > >>>> >>>>> > >>>> >>>>> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/frontends/stream/data_structures.pyc > >>>> >>>>> in __init__(self, pf, dataset_type) > >>>> >>>>> 942 def __init__(self, pf, dataset_type = None): > >>>> >>>>> 943 self.stream_handler = pf.stream_handler > >>>> >>>>> --> 944 super(StreamParticleIndex, self).__init__(pf, > >>>> >>>>> dataset_type) > >>>> >>>>> 945 > >>>> >>>>> 946 def _setup_data_io(self): > >>>> >>>>> > >>>> >>>>> > >>>> >>>>> > >>>> >>>>> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/geometry/particle_geometry_handler.pyc > >>>> >>>>> in __init__(self, pf, dataset_type) > >>>> >>>>> 48 self.directory = > >>>> >>>>> os.path.dirname(self.index_filename) > >>>> >>>>> 49 self.float_type = np.float64 > >>>> >>>>> ---> 50 super(ParticleIndex, self).__init__(pf, > >>>> >>>>> dataset_type) > >>>> >>>>> 51 > >>>> >>>>> 52 def _setup_geometry(self): > >>>> >>>>> > >>>> >>>>> > >>>> >>>>> > >>>> >>>>> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/geometry/geometry_handler.pyc > >>>> >>>>> in __init__(self, pf, dataset_type) > >>>> >>>>> 54 > >>>> >>>>> 55 mylog.debug("Setting up domain geometry.") > >>>> >>>>> ---> 56 self._setup_geometry() > >>>> >>>>> 57 > >>>> >>>>> 58 mylog.debug("Initializing data grid data IO") > >>>> >>>>> > >>>> >>>>> > >>>> >>>>> > >>>> >>>>> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/geometry/particle_geometry_handler.pyc > >>>> >>>>> in _setup_geometry(self) > >>>> >>>>> 52 def _setup_geometry(self): > >>>> >>>>> 53 mylog.debug("Initializing Particle Geometry > >>>> >>>>> Handler.") > >>>> >>>>> ---> 54 self._initialize_particle_handler() > >>>> >>>>> 55 > >>>> >>>>> 56 > >>>> >>>>> > >>>> >>>>> > >>>> >>>>> > >>>> >>>>> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/geometry/particle_geometry_handler.pyc > >>>> >>>>> in _initialize_particle_handler(self) > >>>> >>>>> 87 pf.domain_left_edge, > >>>> >>>>> pf.domain_right_edge, > >>>> >>>>> 88 [N, N, N], len(self.data_files)) > >>>> >>>>> ---> 89 self._initialize_indices() > >>>> >>>>> 90 self.oct_handler.finalize() > >>>> >>>>> 91 self.max_level = self.oct_handler.max_level > >>>> >>>>> > >>>> >>>>> > >>>> >>>>> > >>>> >>>>> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/geometry/particle_geometry_handler.pyc > >>>> >>>>> in _initialize_indices(self) > >>>> >>>>> 109 npart = > >>>> >>>>> sum(data_file.total_particles.values()) > >>>> >>>>> 110 morton[ind:ind + npart] = \ > >>>> >>>>> --> 111 self.io._initialize_index(data_file, > >>>> >>>>> self.regions) > >>>> >>>>> 112 ind += npart > >>>> >>>>> 113 morton.sort() > >>>> >>>>> > >>>> >>>>> > >>>> >>>>> > >>>> >>>>> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/frontends/stream/io.pyc in > >>>> >>>>> _initialize_index(self, data_file, regions) > >>>> >>>>> 144 raise YTDomainOverflow(pos.min(axis=0), > >>>> >>>>> pos.max(axis=0), > >>>> >>>>> 145 > >>>> >>>>> data_file.pf.domain_left_edge, > >>>> >>>>> --> 146 > >>>> >>>>> data_file.pf.domain_right_edge) > >>>> >>>>> 147 regions.add_data_file(pos, > >>>> >>>>> data_file.file_id) > >>>> >>>>> 148 morton.append(compute_morton( > >>>> >>>>> > >>>> >>>>> YTDomainOverflow: Particle bounds [ 0. 0. 0.] and [ > >>>> >>>>> 99.99999237 > >>>> >>>>> 99.99999237 99.99999237] exceed domain bounds [ 0. 0. 0.] > >>>> >>>>> code_length and > >>>> >>>>> [ 1. 1. 1.] code_length > >>>> >>>>> > >>>> >>>>> > >>>> >>>>> > >>>> >>>>> On Thu, Jun 5, 2014 at 8:22 PM, Nathan Goldbaum > >>>> >>>>> <nathan12343@gmail.com> wrote: > >>>> >>>>>> > >>>> >>>>>> Here's a worked out example that does what you're looking for > >>>> >>>>>> using a fake 1 million particle dataset: > >>>> >>>>>> > >>>> >>>>>> > >>>> >>>>>> http://nbviewer.ipython.org/gist/ngoldbaum/546d37869aafe71cfe38 > >>>> >>>>>> > >>>> >>>>>> In this notebook I make use of two key yt features: > >>>> >>>>>> `load_particles`, and `covering_grid`. > >>>> >>>>>> > >>>> >>>>>> load_particles creates a "stream" dataset based on in-memory > >>>> >>>>>> data > >>>> >>>>>> fed in as a numpy array. This dataset acts just like an > >>>> >>>>>> on-disk simulation > >>>> >>>>>> dataset, but doesn't come with the baggage of needing to write > >>>> >>>>>> a custom > >>>> >>>>>> frontend to read a specific data format off disk. > >>>> >>>>>> > >>>> >>>>>> covering_grid is a way to generate uniform resolution data > >>>> >>>>>> from > >>>> >>>>>> an AMR dataset. It acts like a python dictionary where the > >>>> >>>>>> keys are field > >>>> >>>>>> names and returns 3D numpy arrays of whatever uniform > >>>> >>>>>> resolution you specify > >>>> >>>>>> when you create the covering_grid. > >>>> >>>>>> > >>>> >>>>>> Note that if you're using load_particles all of your data > >>>> >>>>>> needs > >>>> >>>>>> to live in memory. If your data is too big for that you'll > >>>> >>>>>> need to write a > >>>> >>>>>> frontend for your data format or use a memmap to an on-disk > >>>> >>>>>> file somehow. > >>>> >>>>>> I'm not an expert on that but others on the list should be > >>>> >>>>>> able to help out. > >>>> >>>>>> > >>>> >>>>>> Hope that gets you well on your way :) > >>>> >>>>>> > >>>> >>>>>> -Nathan > >>>> >>>>>> > >>>> >>>>>> > >>>> >>>>>> On Thu, Jun 5, 2014 at 5:04 PM, Desika Narayanan > >>>> >>>>>> <dnarayan@haverford.edu> wrote: > >>>> >>>>>>> > >>>> >>>>>>> Hey Brendan, > >>>> >>>>>>> > >>>> >>>>>>> A couple of extra tools you might find helpful in conjunction > >>>> >>>>>>> with Nathan's example of depositing the particles onto an > >>>> >>>>>>> octree are at: > >>>> >>>>>>> > >>>> >>>>>>> http://paste.yt-project.org/show/4737/ > >>>> >>>>>>> > >>>> >>>>>>> Where I load a gadget snapshot, and then recover the > >>>> >>>>>>> coordinates > >>>> >>>>>>> and width of each cell. > >>>> >>>>>>> > >>>> >>>>>>> In response to your last question - the particles are > >>>> >>>>>>> deposited > >>>> >>>>>>> into an octree grid (so, you'll see that the cell sizes > >>>> >>>>>>> aren't all the same > >>>> >>>>>>> size). I don't know if depositing onto a regular NxNxN mesh > >>>> >>>>>>> is possible, > >>>> >>>>>>> though would be interested to hear if so. > >>>> >>>>>>> > >>>> >>>>>>> -d > >>>> >>>>>>> > >>>> >>>>>>> > >>>> >>>>>>> On Thu, Jun 5, 2014 at 7:58 PM, Brendan Griffen > >>>> >>>>>>> <brendan.f.griffen@gmail.com> wrote: > >>>> >>>>>>>> > >>>> >>>>>>>> Thanks. I'll get the "bleeding edge" version first then try > >>>> >>>>>>>> your suggestions. Though I want to return the NxNxN array > >>>> >>>>>>>> and be able to > >>>> >>>>>>>> write this mesh to a file. It is *only* using the cic part > >>>> >>>>>>>> of yt and it > >>>> >>>>>>>> should return the mesh to be written? Just wanted to > >>>> >>>>>>>> clarify? > >>>> >>>>>>>> > >>>> >>>>>>>> Thanks. > >>>> >>>>>>>> Brendan > >>>> >>>>>>>> > >>>> >>>>>>>> > >>>> >>>>>>>> On Thu, Jun 5, 2014 at 6:49 PM, Nathan Goldbaum > >>>> >>>>>>>> <nathan12343@gmail.com> wrote: > >>>> >>>>>>>>> > >>>> >>>>>>>>> > >>>> >>>>>>>>> > >>>> >>>>>>>>> > >>>> >>>>>>>>> On Thu, Jun 5, 2014 at 3:36 PM, John ZuHone > >>>> >>>>>>>>> <jzuhone@gmail.com> wrote: > >>>> >>>>>>>>>> > >>>> >>>>>>>>>> Hi Brendan, > >>>> >>>>>>>>>> > >>>> >>>>>>>>>> Which version of yt are you using? > >>>> >>>>>>>>>> > >>>> >>>>>>>>>> If you're using 3.0, this is actually fairly easy. If you > >>>> >>>>>>>>>> look in yt.fields.particle_fields.py, around line 85, you > >>>> >>>>>>>>>> can see how this > >>>> >>>>>>>>>> is done for the "particle_density" and "particle_mass" > >>>> >>>>>>>>>> fields. Basically you > >>>> >>>>>>>>>> can call a "deposit" method which takes the particle field > >>>> >>>>>>>>>> quantity you want > >>>> >>>>>>>>>> deposited and deposits it into cells. The underlying > >>>> >>>>>>>>>> calculation is done > >>>> >>>>>>>>>> using Cython, so it's fast. > >>>> >>>>>>>>> > >>>> >>>>>>>>> > >>>> >>>>>>>>> And you shouldn't ever actually need to call these > >>>> >>>>>>>>> "deposit" > >>>> >>>>>>>>> functions, since "deposit" is exposed as a field type for > >>>> >>>>>>>>> all datasets that > >>>> >>>>>>>>> contain particles. > >>>> >>>>>>>>> > >>>> >>>>>>>>> Here is a notebook that does this for Enzo AMR data: > >>>> >>>>>>>>> > >>>> >>>>>>>>> > >>>> >>>>>>>>> > >>>> >>>>>>>>> http://nbviewer.ipython.org/gist/ngoldbaum/5e19e4e6cc2bf330149c > >>>> >>>>>>>>> > >>>> >>>>>>>>> This dataset contains about a million particles and > >>>> >>>>>>>>> generates > >>>> >>>>>>>>> a CIC deposition for the whole domain in about 6 seconds > >>>> >>>>>>>>> from a cold start. > >>>> >>>>>>>>> > >>>> >>>>>>>>>> > >>>> >>>>>>>>>> > >>>> >>>>>>>>>> If you're using 2.x, then you can do the same thing, but > >>>> >>>>>>>>>> it's > >>>> >>>>>>>>>> not as straightforward. You can see how this works in > >>>> >>>>>>>>>> yt.data_objects.universal_fields.py, around line 986, > >>>> >>>>>>>>>> where the > >>>> >>>>>>>>>> "particle_density" field is defined. Basically, it calls > >>>> >>>>>>>>>> CICDeposit_3, which > >>>> >>>>>>>>>> is in yt.utilities.lib.CICDeposit.pyx. > >>>> >>>>>>>>>> > >>>> >>>>>>>>>> Let me know if you need any more clarification. > >>>> >>>>>>>>>> > >>>> >>>>>>>>>> Best, > >>>> >>>>>>>>>> > >>>> >>>>>>>>>> John Z > >>>> >>>>>>>>>> > >>>> >>>>>>>>>> On Jun 5, 2014, at 6:07 PM, Brendan Griffen > >>>> >>>>>>>>>> <brendan.f.griffen@gmail.com> wrote: > >>>> >>>>>>>>>> > >>>> >>>>>>>>>> > Hi, > >>>> >>>>>>>>>> > > >>>> >>>>>>>>>> > I was wondering if there were any Cython routines within > >>>> >>>>>>>>>> > yt > >>>> >>>>>>>>>> > which takes particle data and converts it into a > >>>> >>>>>>>>>> > cloud-in-cell based mesh > >>>> >>>>>>>>>> > which can be written to a file of my choosing. > >>>> >>>>>>>>> > >>>> >>>>>>>>> > >>>> >>>>>>>>> What sort of mesh were you looking for? yt will internally > >>>> >>>>>>>>> construct an octree if it is fed particle data. I'm not > >>>> >>>>>>>>> sure whether this > >>>> >>>>>>>>> octree can be saved to disk for later analysis. > >>>> >>>>>>>>> > >>>> >>>>>>>>> It's also possible to create a uniform resolution covering > >>>> >>>>>>>>> grid containing field data for a deposited quantity, which > >>>> >>>>>>>>> can be quite > >>>> >>>>>>>>> easily saved to disk in a number of ways. > >>>> >>>>>>>>> > >>>> >>>>>>>>>> > >>>> >>>>>>>>>> I heard a while ago there was some such functionality but > >>>> >>>>>>>>>> it > >>>> >>>>>>>>>> could be too far down the yt rabbit hole to be used as a > >>>> >>>>>>>>>> standalone? Is this > >>>> >>>>>>>>>> true? I have my own Python code for doing it but it just > >>>> >>>>>>>>>> isn't fast enough > >>>> >>>>>>>>>> and thought I'd ask the yt community if there were any > >>>> >>>>>>>>>> wrapper tools > >>>> >>>>>>>>>> available to boost the speed. > >>>> >>>>>>>>>> > > >>>> >>>>>>>>>> > Thanks. > >>>> >>>>>>>>>> > Brendan > >>>> >>>>>>>>>> > _______________________________________________ > >>>> >>>>>>>>>> > yt-users mailing list > >>>> >>>>>>>>>> > yt-users@lists.spacepope.org > >>>> >>>>>>>>>> > > >>>> >>>>>>>>>> > > >>>> >>>>>>>>>> > http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org > >>>> >>>>>>>>>> > >>>> >>>>>>>>>> _______________________________________________ > >>>> >>>>>>>>>> yt-users mailing list > >>>> >>>>>>>>>> yt-users@lists.spacepope.org > >>>> >>>>>>>>>> > >>>> >>>>>>>>>> > >>>> >>>>>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org > >>>> >>>>>>>>> > >>>> >>>>>>>>> > >>>> >>>>>>>>> > >>>> >>>>>>>>> _______________________________________________ > >>>> >>>>>>>>> yt-users mailing list > >>>> >>>>>>>>> yt-users@lists.spacepope.org > >>>> >>>>>>>>> > >>>> >>>>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org > >>>> >>>>>>>>> > >>>> >>>>>>>> > >>>> >>>>>>>> > >>>> >>>>>>>> _______________________________________________ > >>>> >>>>>>>> yt-users mailing list > >>>> >>>>>>>> yt-users@lists.spacepope.org > >>>> >>>>>>>> > >>>> >>>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org > >>>> >>>>>>>> > >>>> >>>>>>> > >>>> >>>>>>> > >>>> >>>>>>> _______________________________________________ > >>>> >>>>>>> yt-users mailing list > >>>> >>>>>>> yt-users@lists.spacepope.org > >>>> >>>>>>> > >>>> >>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org > >>>> >>>>>>> > >>>> >>>>>> > >>>> >>>>>> > >>>> >>>>>> _______________________________________________ > >>>> >>>>>> yt-users mailing list > >>>> >>>>>> yt-users@lists.spacepope.org > >>>> >>>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org > >>>> >>>>>> > >>>> >>>>> > >>>> >>>>> > >>>> >>>>> _______________________________________________ > >>>> >>>>> yt-users mailing list > >>>> >>>>> yt-users@lists.spacepope.org > >>>> >>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org > >>>> >>>>> > >>>> >>>> > >>>> >>>> > >>>> >>>> _______________________________________________ > >>>> >>>> yt-users mailing list > >>>> >>>> yt-users@lists.spacepope.org > >>>> >>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org > >>>> >>>> > >>>> >>> > >>>> >>> > >>>> >>> _______________________________________________ > >>>> >>> yt-users mailing list > >>>> >>> yt-users@lists.spacepope.org > >>>> >>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org > >>>> >>> > >>>> >> > >>>> >> > >>>> >> _______________________________________________ > >>>> >> yt-users mailing list > >>>> >> yt-users@lists.spacepope.org > >>>> >> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org > >>>> >> > >>>> > > >>>> > > >>>> > _______________________________________________ > >>>> > yt-users mailing list > >>>> > yt-users@lists.spacepope.org > >>>> > http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org > >>>> > > >>>> > >>>> > >>>> _______________________________________________ > >>>> yt-users mailing list > >>>> yt-users@lists.spacepope.org > >>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org > >>>> > >>> > >> > >> > >> _______________________________________________ > >> yt-users mailing list > >> yt-users@lists.spacepope.org > >> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org > >> > > > > > > _______________________________________________ > > yt-users mailing list > > yt-users@lists.spacepope.org > > http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org > > > _______________________________________________ > yt-users mailing list > yt-users@lists.spacepope.org > http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
Yes, Matt and I are chatting off-list about this. I'll have more to say later hopefully. On Mon, Jun 9, 2014 at 10:48 AM, Brendan Griffen < brendan.f.griffen@gmail.com> wrote:
Hi Nathan, I'm not sure if you missed it but earlier Matt asked if there was some copying which shouldn't be happening?
Brendan
On Mon, Jun 9, 2014 at 1:22 PM, Nathan Goldbaum <nathan12343@gmail.com> wrote:
On Mon, Jun 9, 2014 at 12:32 AM, Brendan Griffen < brendan.f.griffen@gmail.com> wrote:
Thanks Nathan. Comparing the two though:
26 1536.664 MiB 1130.480 MiB uniform_array = grid_object['deposit', 'all_cic']
24 1285.703 MiB 890.539 MiB cic_density = ad["deposit", "all_cic"]
Isn't it the uniform grid which uses the most memory? I'm running the profiling now and will get back to you. I hope in spite of the fact that it does crash it will still give me some useful output.
That's true - I was thinking it might be more memory efficient since there is no need to construct the covering grid in addition to the octree.
Brendan
On Mon, Jun 9, 2014 at 2:05 AM, Nathan Goldbaum <nathan12343@gmail.com> wrote:
Hey Brendan,
Could you try running your script using the memory_profiler module on pypi?
Here's an example script that uses the memory profiler: http://paste.yt-project.org/show/4748/
and the output for that script: http://paste.yt-project.org/show/4749/
For what it's worth, it does indeed look like matt's suggestion to use load_uniform_grid is an option, and might be more memory efficient in the end since you will go directly to the uniform grid you want without creating an octree. Here's an example: http://paste.yt-project.org/show/4750/
Here's the memory usage information for that example: http://paste.yt-project.org/show/4751/
I used a 256^3 uniform grid with normally distributed random data - I'm not sure whether it will also be more memory efficient in your case.
On Sun, Jun 8, 2014 at 9:32 PM, Brendan Griffen < brendan.f.griffen@gmail.com> wrote:
This is the full error if it helps at all? It is indeed, loading in all of the quantities.
Loading particles... --> Loading particle type: 1 yt : [INFO ] 2014-06-08 15:12:05,540 Parameters: current_time = 0.0 yt : [INFO ] 2014-06-08 15:12:05,540 Parameters: domain_dimensions = [2 2 2] yt : [INFO ] 2014-06-08 15:12:05,541 Parameters: domain_left_edge = [ 0. 0. 0.] yt : [INFO ] 2014-06-08 15:12:05,542 Parameters: domain_right_edge = [ 100. 100. 100.] yt : [INFO ] 2014-06-08 15:12:05,542 Parameters: cosmological_simulation = 0.0 yt : [INFO ] 2014-06-08 15:12:05,548 Allocating for 1.074e+09 particles yt : [INFO ] 2014-06-08 15:16:16,195 Identified 7.584e+07 octs yt : [INFO ] 2014-06-08 15:16:16,299 Loading field plugins. yt : [INFO ] 2014-06-08 15:16:16,299 Loaded angular_momentum (8 new fields) yt : [INFO ] 2014-06-08 15:16:16,299 Loaded astro (14 new fields) yt : [INFO ] 2014-06-08 15:16:16,300 Loaded cosmology (20 new fields) yt : [INFO ] 2014-06-08 15:16:16,300 Loaded fluid (56 new fields) yt : [INFO ] 2014-06-08 15:16:16,301 Loaded fluid_vector (88 new fields) yt : [INFO ] 2014-06-08 15:16:16,301 Loaded geometric (103 new fields) yt : [INFO ] 2014-06-08 15:16:16,301 Loaded local (103 new fields) yt : [INFO ] 2014-06-08 15:16:16,302 Loaded magnetic_field (109 new fields) yt : [INFO ] 2014-06-08 15:16:16,302 Loaded species (109 new fields)
--------------------------------------------------------------------------- MemoryError Traceback (most recent call last) /nfs/blank/h4231/bgriffen/data/lib/yt-x86_64/lib/python2.7/site-packages/IPython/utils/py3compat.pyc in execfile(fname, *where) 202 else: 203 filename = fname --> 204 __builtin__.execfile(filename, *where)
/nfs/blank/h4231/bgriffen/work/projects/caterpillar/c2ray/cic/ytcic.py in <module>() 99 slc.set_figure_size(4) 100 slc.save() --> 101 102 for ndim in dimlist: 103 print
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc in __getitem__(self, key) 218 return self.field_data[f] 219 else: --> 220 self.get_data(f) 221 # fi.units is the unit expression string. We depend on the registry 222 # hanging off the dataset to define this unit object.
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc in get_data(self, fields) 627 628 fields_to_generate += gen_fluids + gen_particles --> 629 self._generate_fields(fields_to_generate) 630 631 def _generate_fields(self, fields_to_generate):
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc in _generate_fields(self, fields_to_generate) 644 fi = self.pf._get_field_info(*field) 645 try: --> 646 fd = self._generate_field(field) 647 if type(fd) == np.ndarray: 648 fd = self.pf.arr(fd, fi.units)
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc in _generate_field(self, field) 255 tr = self._generate_particle_field(field) 256 else: --> 257 tr = self._generate_fluid_field(field) 258 if tr is None: 259 raise YTCouldNotGenerateField(field, self.pf)
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc in _generate_fluid_field(self, field) 273 finfo.check_available(gen_obj) 274 except NeedsGridType as ngt_exception: --> 275 rv = self._generate_spatial_fluid(field, ngt_exception.ghost_zones) 276 else: 277 rv = finfo(gen_obj)
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc in _generate_spatial_fluid(self, field, ngz) 289 o = self._current_chunk.objs[0] 290 with o._activate_cache(): --> 291 ind += o.select(self.selector, self[field], rv, ind) 292 else: 293 chunks = self.index._chunk(self, "spatial", ngz = ngz)
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc in __getitem__(self, key) 218 return self.field_data[f] 219 else: --> 220 self.get_data(f) 221 # fi.units is the unit expression string. We depend on the registry 222 # hanging off the dataset to define this unit object.
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc in get_data(self, fields) 627 628 fields_to_generate += gen_fluids + gen_particles --> 629 self._generate_fields(fields_to_generate) 630 631 def _generate_fields(self, fields_to_generate):
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc in _generate_fields(self, fields_to_generate) 644 fi = self.pf._get_field_info(*field) 645 try: --> 646 fd = self._generate_field(field) 647 if type(fd) == np.ndarray: 648 fd = self.pf.arr(fd, fi.units)
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc in _generate_field(self, field) 255 tr = self._generate_particle_field(field) 256 else: --> 257 tr = self._generate_fluid_field(field) 258 if tr is None: 259 raise YTCouldNotGenerateField(field, self.pf)
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc in _generate_fluid_field(self, field) 275 rv = self._generate_spatial_fluid(field, ngt_exception.ghost_zones) 276 else: --> 277 rv = finfo(gen_obj) 278 return rv 279
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/fields/derived_field.pyc in __call__(self, data) 176 "for %s" % (self.name,)) 177 with self.unit_registry(data): --> 178 dd = self._function(self, data) 179 for field_name in data.keys(): 180 if field_name not in original_fields:
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/fields/particle_fields.pyc in particle_cic(field, data) 113 def particle_cic(field, data): 114 pos = data[ptype, coord_name] --> 115 d = data.deposit(pos, [data[ptype, mass_name]], method = "cic") 116 d = data.apply_units(d, data[ptype, mass_name].units) 117 d /= data["index", "cell_volume"]
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/octree_subset.pyc in deposit(self, positions, fields, method) 167 mylog.debug("Depositing %s (%s^3) particles into %s Octs", 168 positions.shape[0], positions.shape[0]**0.3333333, nvals[-1]) --> 169 pos = np.asarray(positions.convert_to_units("code_length"), 170 dtype="float64") 171 # We should not need the following if we know in advance all our fields
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/units/yt_array.pyc in convert_to_units(self, units) 366 367 self.units = new_units --> 368 self *= conversion_factor 369 return self 370
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/units/yt_array.pyc in __imul__(self, other) 667 """ See __mul__. """ 668 oth = sanitize_units_mul(self, other) --> 669 return np.multiply(self, oth, out=self) 670 671 def __div__(self, right_object):
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/units/yt_array.pyc in __array_wrap__(self, out_arr, context) 966 # casting to YTArray avoids creating a YTQuantity with size > 1 967 return YTArray(np.array(out_arr, unit)) --> 968 return ret_class(np.array(out_arr), unit) 969 970
MemoryError:
On Sun, Jun 8, 2014 at 10:50 PM, Matthew Turk <matthewturk@gmail.com> wrote:
Hi Brendan,
On Sun, Jun 8, 2014 at 9:21 PM, Brendan Griffen <brendan.f.griffen@gmail.com> wrote: > Hi Matt, > > Thanks for your detailed email. Forgive my naivety but why do you need the > oct-tree in the first place? I have a my own fortran code for constructing a > cloud in cell mesh and it uses very little overhead (just the n^3 grid and > the particle data itself). I then calculate the dx,dy,dzs to the nearest 8 > grid points and distribute accordingly in a omp loop which is done in a > fraction of a second. Does the situation with yt come about (oct tree etc.) > necessarily because of the way it handles particle data? Is it essentially > used to map the particles to domains in the grid or something?
That's not naive at all. There are two reasons --
1) The octree is used for indexing for neighbor lookups and early-termination of region selection for particles 2) The octree is used to estimate the "required resolution" for any operation that requires a space-filling value. (i.e., any time that a particle becomes a volume.)
Projections in yt are adaptive, in that they project down to the finest appropriate resolution. There's also the "arbitrary_grid" operation, which does precisely what you're describing, but as it stands right now the octree gets constructed at time of instantiation of the indexing system. Thinking it over, you may be able to avoid that completely by not using load_particles and instead using load_uniform_grid and supplying your desired dimensions. The field names should be the same.
> > The machine I max memory on has 128GB and the snapshots are using 1024^3 > particles. Do you have any idea of how much memory the oct-tree uses as a > function of particle/grid number? I am going to try on a 256GB machine > (though this is a bit of a hassle). I'll see how I go.
I am disappointed that it's blowing out your RAM. This week I will try to get some memory profiling done. Could you file a bug to this effect, which will help me track it? Peak memory usage during indexing should only be 64 bits * Nparticles, unless you're using load_particles, in which case all the fields will *also* have to be in memory. It's about 8 gigabytes per field. So, I think there's something going wrong.
> > Thanks. > > Brendan > > > On Sun, Jun 8, 2014 at 6:25 PM, Matthew Turk <matthewturk@gmail.com> wrote: >> >> Hi all, >> >> I feel like I owe a brief explanation of why things are tricky right >> now, what we're planning on doing, and how we're experimenting and >> developing. >> >> Presently, the particle geometry handlers build a single mesh from all >> particles in the dataset, along with a coarse bitmask that correlates >> files to regions in the domain. This requires the allocation of a >> single int64 array of size Nparticles, which is sorted in place and >> then fed into an octree construction algorithm that then spits back >> out the mesh. Each octree component contains 3 64-bit integers and >> eitehr a void pointer or a pointer to eight other octs. Increasing >> n_ref decreases the number of octs in this mesh; when smoothing >> operaitons are conducted, a second "index" mesh is created for looking >> up particles near mesh points. Mesh points are used for adaptive >> resolution smoothing and other "deposit particles on the grid somehow" >> operations (including SPH kernel). >> >> Anyway, because right now it requires a global mesh to be constructed, >> this is expensive and requires holding a 64-bit integer in memory for >> each particle. I think if you're loading the particles in differently >> there is some additional overhead as well, but I'm still a bit >> surprised you OOM on a 1024^3 dataset. >> >> In general, we don't *need* this global mesh; is can be constructed as >> required, which would speed up both the initial index phase as well as >> the final meshing process. I got about 50% of the way to implementing >> this last fall, but because of various concerns and deadlines I >> haven't finished it. I intend to get back to it probably in July, >> right after we put out a 3.0, so that we can have it in time for 3.1. >> In principle this will make the particle codes much more similar to >> ARTIO, in that the mesh will be constructed only as required and >> discarded when no longer required, which will make them much more >> memory efficient. >> >> But, getting a single mesh for extremely large data is a very high >> priority; right now for the 10240^3 run we've been loading up >> individual sub-chunks, which I want to stop doing. >> >> From the technical perspective, these are the things that need to >> happen on the yt side for particle datasets to move to this "lazy" >> mode of loading; most of this is based on things learned from 2HOT and >> ARTIO, and will involve converting to a forest-of-octrees. >> >> * Split into spatially-organized subchunks of ParticleOctreeSubset >> objects, such that these map 1:Nfiles, and that can be constructed on >> the fly. >> * Construct a dual-mesh of the bitmask "ParticleRegion" object that >> will help with identifying neighbors to a given oct cell, so that if >> we're inside one octree we know which neighbor octrees to grab if we >> need particles for smoothing things (fast boundary particle >> identification is later down the road) >> * Parallel sort of particles, or using the parallel ring function; >> may not be necessary after all >> >> All of this is doable, and I'd be happy to work with people if they'd >> like to take a shot at implementing it, but I've mostly put it on my >> list for post-3.0. >> >> -Matt >> >> On Sun, Jun 8, 2014 at 2:43 PM, Nathan Goldbaum < nathan12343@gmail.com> >> wrote: >> > >> > >> > >> > On Sun, Jun 8, 2014 at 12:27 PM, Brendan Griffen >> > <brendan.f.griffen@gmail.com> wrote: >> >> >> >> Also, how do I construct just a zero filled yt array with dimensions >> >> (ndim,ndim,ndim)? Thanks >> > >> > >> > >> > from yt import YTArray >> > from numpy import np >> > >> > arr = YTArray(np.zeros([ndim, ndim, ndim]), input_units=units_string) >> > >> > or alternatively: >> > >> > from yt.units import kiloparsec >> > >> > arr = kiloparsec*np.zeros([ndim, ndim, ndim]) >> > >> > it doesn't have to be kiloparsec - you can compose the units you want >> > out of >> > any of the unit symbols that live in yt.units. >> > >> > See this page for a ton more detail about yt's new unit system: >> > http://yt-project.org/docs/dev-3.0/analyzing/units/index.html >> > >> >> >> >> >> >> Brendan >> >> >> >> >> >> On Sun, Jun 8, 2014 at 3:26 PM, Brendan Griffen >> >> <brendan.f.griffen@gmail.com> wrote: >> >>> >> >>> Hi, >> >>> >> >>> Since I get memory errors. Could I not just read in the blocks of the >> >>> output individually then basically stack the mesh each time. That way >> >>> not >> >>> every single particle of the snapshot has to be loaded at the same >> >>> time. >> >>> Would that just be a case of doing >> >>> >> >>> level = int(math.log(ndim,2)) >> >>> cg = ds.covering_grid(level=level, >> >>> left_edge=[0,0,0],dims=[ndim,ndim,ndim]) >> >>> arr = cg['deposit', 'all_density'] >> >>> arrall += arr >> >>> >> >>> in a loop over each HDF5 block? >> > >> > >> > It's likely that the memory use is dominated by the octree rather than >> > the >> > covering grid. This is with 1024^3 particles, correct? >> > >> > You can probably significantly reduce the memory used by the octree by >> > increasing n_ref in the call to load_particles. >> > >> > See this page for more detail about load_particles: >> > >> > http://yt-project.org/docs/dev-3.0/examining/loading_data.html#generic-parti... >> > >> > Larger n_ref means fewer octree cells (lower resolution), but it also >> > means >> > lower poisson noise and lower memory use. >> > >> > Alternatively, as Matt suggested, you could break your 1024^3 ensemble >> > of >> > particles up into chunks, loop over the chunk, creating a particle >> > octree >> > and then a covering grid for each subset of the particles. Your final >> > covering grid is just the sub of the covering grids for each subset of >> > particles. >> > >> >>> >> >>> >> >>> Thanks. >> >>> Brendan >> >>> >> >>> >> >>> >> >>> >> >>> On Fri, Jun 6, 2014 at 7:26 PM, Matthew Turk < matthewturk@gmail.com> >> >>> wrote: >> >>>> >> >>>> >> >>>> On Jun 6, 2014 4:54 PM, "Brendan Griffen" >> >>>> <brendan.f.griffen@gmail.com> >> >>>> wrote: >> >>>> > >> >>>> > OK great. It is very low resolution but it worked. Thanks for all >> >>>> > your >> >>>> > help. My higher resolution run 1024^3 in 100 Mpc seems to crash on >> >>>> > 128GB >> >>>> > memory machine. I might have to look elsewhere. >> >>>> > >> >>>> >> >>>> If you are looking for low resolution extraction you can tune the >> >>>> memory usage by changing the parameter n_ref to something higher. >> >>>> >> >>>> Supporting extremely large datasets in a single mesh is on the >> >>>> roadmap >> >>>> for the late summer or fall, after a 3.0 release goes out. For now >> >>>> you can >> >>>> also extract before you load in; this is sort of how we are >> >>>> supporting an >> >>>> INCITE project with very large particle counts. >> >>>> >> >>>> >> >>>> > Also, I normally use Canopy distribution but I just use an alias to >> >>>> > loadyt which erases my PYTHONPATH and I can't access scipy and a >> >>>> > few other >> >>>> > libraries any more. What is the best practice here? Should I just >> >>>> > manually >> >>>> > export PYTHONPATH and point to the libraries need in canopy or can >> >>>> > they play >> >>>> > nice together? >> >>>> > >> >>>> > Thanks. >> >>>> > >> >>>> > BG >> >>>> > >> >>>> > >> >>>> > On Fri, Jun 6, 2014 at 2:54 PM, Nathan Goldbaum >> >>>> > <nathan12343@gmail.com> wrote: >> >>>> >> >> >>>> >> >> >>>> >> >> >>>> >> >> >>>> >> On Fri, Jun 6, 2014 at 11:48 AM, Brendan Griffen >> >>>> >> <brendan.f.griffen@gmail.com> wrote: >> >>>> >>> >> >>>> >>> OK great. Thanks. I just wanted a homogeneous mesh. 512^3 with no >> >>>> >>> nesting of any kind. Though when I plot the image it looks like >> >>>> >>> it is >> >>>> >>> assigning particles incorrectly (low resolution on the outside). >> >>>> >>> This is >> >>>> >>> just a test image. >> >>>> >>> >> >>>> >> >> >>>> >> The SlicePlot is visualizing the octree so there is less >> >>>> >> resolution >> >>>> >> where there are fewer particles. If you want to visualize the >> >>>> >> covering grid >> >>>> >> you're going to need to visualize that separately. >> >>>> >> >> >>>> >>> >> >>>> >>> ds = yt.load_particles(data, length_unit=3.08e24, >> >>>> >>> mass_unit=1.9891e33,bbox=bbox) >> >>>> >>> >> >>>> >>> ad = ds.all_data() >> >>>> >>> print ad['deposit', 'all_cic'] >> >>>> >>> slc = yt.SlicePlot(ds, 2, ('deposit', 'all_cic')) >> >>>> >>> slc.set_figure_size(4) >> >>>> >>> cg = ds.covering_grid(level=9, >> >>>> >>> left_edge=[0,0,0],dims=[512,512,512]) >> >>>> >>> >> >>>> >> >> >>>> >> To actually produce the uniform resolution ndarray, you're going >> >>>> >> to >> >>>> >> need to do something like: >> >>>> >> >> >>>> >> array = cg[('deposit', 'all_cic')] >> >>>> >> >> >>>> >> array will then be a 3D array you can do whatever you want with. >> >>>> >> By >> >>>> >> default it has units, but to strip them off you'll just need to >> >>>> >> cast to >> >>>> >> ndarray: >> >>>> >> >> >>>> >> array_without_units = array.v >> >>>> >> >> >>>> >> >> >>>> >>> >> >>>> >>> Also, is there a way to load multiple particle types? >> >>>> >>> >> >>>> >>> Do I just need to stack the particles into the array here? >> >>>> >>> >> >>>> >>> data = {'particle_position_x': pos[:,0], >> >>>> >>> 'particle_position_y': pos[:,1], >> >>>> >>> 'particle_position_z': pos[:,2], >> >>>> >>> 'particle_mass': np.array([mpart]*npart)} >> >>>> >>> >> >>>> >>> Then feed it in as usual? >> >>>> >> >> >>>> >> >> >>>> >> That's right, although if the particle masses are different for >> >>>> >> the >> >>>> >> different particle types that code snippet will need to be >> >>>> >> generalized to >> >>>> >> handle that. >> >>>> >> >> >>>> >> I think in principle it should be possible to make load_particles >> >>>> >> handle different particle types just like an SPH dataset that >> >>>> >> contains >> >>>> >> multiple particle types, but right now that hasn't been >> >>>> >> implemented yet. >> >>>> >> >> >>>> >>> >> >>>> >>> >> >>>> >>> Brendan >> >>>> >>> >> >>>> >>> >> >>>> >>> On Thu, Jun 5, 2014 at 9:44 PM, Nathan Goldbaum >> >>>> >>> <nathan12343@gmail.com> wrote: >> >>>> >>>> >> >>>> >>>> That's right, you can set that via the bbox keyword parameter >> >>>> >>>> for >> >>>> >>>> load_particles. I'd urge you to take a look at the docstrings >> >>>> >>>> and source >> >>>> >>>> code for load_particles. >> >>>> >>>> >> >>>> >>>> >> >>>> >>>> On Thu, Jun 5, 2014 at 6:34 PM, Brendan Griffen >> >>>> >>>> <brendan.f.griffen@gmail.com> wrote: >> >>>> >>>>> >> >>>> >>>>> Thanks very much Nathan. I tried to load in my own data but I >> >>>> >>>>> think there are too many particles or I have to specifically >> >>>> >>>>> set the domain >> >>>> >>>>> size. >> >>>> >>>>> >> >>>> >>>>> In this area: >> >>>> >>>>> >> >>>> >>>>> data = {'particle_position_x': pos[:,0], >> >>>> >>>>> 'particle_position_y': pos[:,1], >> >>>> >>>>> 'particle_position_z': pos[:,2], >> >>>> >>>>> 'particle_mass': np.array([mpart]*npart)} >> >>>> >>>>> >> >>>> >>>>> ds = yt.load_particles(data, length_unit=3.08e24, >> >>>> >>>>> mass_unit=1.9891e36) >> >>>> >>>>> ad = ds.all_data() >> >>>> >>>>> print ad['deposit', 'all_cic'] >> >>>> >>>>> >> >>>> >>>>> In [3]: run ytcic.py >> >>>> >>>>> yt : [INFO ] 2014-06-05 21:29:06,183 Parameters: >> >>>> >>>>> current_time >> >>>> >>>>> = 0.0 >> >>>> >>>>> yt : [INFO ] 2014-06-05 21:29:06,183 Parameters: >> >>>> >>>>> domain_dimensions = [2 2 2] >> >>>> >>>>> yt : [INFO ] 2014-06-05 21:29:06,184 Parameters: >> >>>> >>>>> domain_left_edge = [ 0. 0. 0.] >> >>>> >>>>> yt : [INFO ] 2014-06-05 21:29:06,185 Parameters: >> >>>> >>>>> domain_right_edge = [ 1. 1. 1.] >> >>>> >>>>> yt : [INFO ] 2014-06-05 21:29:06,185 Parameters: >> >>>> >>>>> cosmological_simulation = 0.0 >> >>>> >>>>> yt : [INFO ] 2014-06-05 21:29:06,188 Allocating for >> >>>> >>>>> 1.342e+08 >> >>>> >>>>> particles >> >>>> >>>>> >> >>>> >>>>> >> >>>> >>>>> --------------------------------------------------------------------------- >> >>>> >>>>> YTDomainOverflow Traceback (most >> >>>> >>>>> recent >> >>>> >>>>> call last) >> >>>> >>>>> >> >>>> >>>>> >> >>>> >>>>> /nfs/blank/h4231/bgriffen/data/lib/yt-x86_64/lib/python2.7/site-packages/IPython/utils/py3compat.pyc >> >>>> >>>>> in execfile(fname, *where) >> >>>> >>>>> 202 else: >> >>>> >>>>> 203 filename = fname >> >>>> >>>>> --> 204 __builtin__.execfile(filename, *where) >> >>>> >>>>> >> >>>> >>>>> >> >>>> >>>>> >> >>>> >>>>> /nfs/blank/h4231/bgriffen/work/projects/caterpillar/c2ray/cic/ytcic.py in >> >>>> >>>>> <module>() >> >>>> >>>>> 52 >> >>>> >>>>> 53 ad = ds.all_data() >> >>>> >>>>> ---> 54 print ad['deposit', 'all_cic'] >> >>>> >>>>> 55 slc = yt.SlicePlot(ds, 2, ('deposit', 'all_cic')) >> >>>> >>>>> 56 slc.set_figure_size(4) >> >>>> >>>>> >> >>>> >>>>> >> >>>> >>>>> >> >>>> >>>>> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc >> >>>> >>>>> in __getitem__(self, key) >> >>>> >>>>> 205 Returns a single field. Will add if necessary. >> >>>> >>>>> 206 """ >> >>>> >>>>> --> 207 f = self._determine_fields([key])[0] >> >>>> >>>>> 208 if f not in self.field_data and key not in >> >>>> >>>>> self.field_data: >> >>>> >>>>> 209 if f in self._container_fields: >> >>>> >>>>> >> >>>> >>>>> >> >>>> >>>>> >> >>>> >>>>> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc >> >>>> >>>>> in _determine_fields(self, fields) >> >>>> >>>>> 453 raise YTFieldNotParseable(field) >> >>>> >>>>> 454 ftype, fname = field >> >>>> >>>>> --> 455 finfo = self.pf._get_field_info(ftype, >> >>>> >>>>> fname) >> >>>> >>>>> 456 else: >> >>>> >>>>> 457 fname = field >> >>>> >>>>> >> >>>> >>>>> >> >>>> >>>>> >> >>>> >>>>> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/static_output.pyc >> >>>> >>>>> in _get_field_info(self, ftype, fname) >> >>>> >>>>> 445 _last_finfo = None >> >>>> >>>>> 446 def _get_field_info(self, ftype, fname = None): >> >>>> >>>>> --> 447 self.index >> >>>> >>>>> 448 if fname is None: >> >>>> >>>>> 449 ftype, fname = "unknown", ftype >> >>>> >>>>> >> >>>> >>>>> >> >>>> >>>>> >> >>>> >>>>> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/static_output.pyc >> >>>> >>>>> in index(self) >> >>>> >>>>> 277 raise RuntimeError("You should not >> >>>> >>>>> instantiate Dataset.") >> >>>> >>>>> 278 self._instantiated_index = >> >>>> >>>>> self._index_class( >> >>>> >>>>> --> 279 self, dataset_type=self.dataset_type) >> >>>> >>>>> 280 # Now we do things that we need an >> >>>> >>>>> instantiated index for >> >>>> >>>>> 281 # ...first off, we create our field_info >> >>>> >>>>> now. >> >>>> >>>>> >> >>>> >>>>> >> >>>> >>>>> >> >>>> >>>>> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/frontends/stream/data_structures.pyc >> >>>> >>>>> in __init__(self, pf, dataset_type) >> >>>> >>>>> 942 def __init__(self, pf, dataset_type = None): >> >>>> >>>>> 943 self.stream_handler = pf.stream_handler >> >>>> >>>>> --> 944 super(StreamParticleIndex, self).__init__(pf, >> >>>> >>>>> dataset_type) >> >>>> >>>>> 945 >> >>>> >>>>> 946 def _setup_data_io(self): >> >>>> >>>>> >> >>>> >>>>> >> >>>> >>>>> >> >>>> >>>>> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/geometry/particle_geometry_handler.pyc >> >>>> >>>>> in __init__(self, pf, dataset_type) >> >>>> >>>>> 48 self.directory = >> >>>> >>>>> os.path.dirname(self.index_filename) >> >>>> >>>>> 49 self.float_type = np.float64 >> >>>> >>>>> ---> 50 super(ParticleIndex, self).__init__(pf, >> >>>> >>>>> dataset_type) >> >>>> >>>>> 51 >> >>>> >>>>> 52 def _setup_geometry(self): >> >>>> >>>>> >> >>>> >>>>> >> >>>> >>>>> >> >>>> >>>>> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/geometry/geometry_handler.pyc >> >>>> >>>>> in __init__(self, pf, dataset_type) >> >>>> >>>>> 54 >> >>>> >>>>> 55 mylog.debug("Setting up domain geometry.") >> >>>> >>>>> ---> 56 self._setup_geometry() >> >>>> >>>>> 57 >> >>>> >>>>> 58 mylog.debug("Initializing data grid data IO") >> >>>> >>>>> >> >>>> >>>>> >> >>>> >>>>> >> >>>> >>>>> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/geometry/particle_geometry_handler.pyc >> >>>> >>>>> in _setup_geometry(self) >> >>>> >>>>> 52 def _setup_geometry(self): >> >>>> >>>>> 53 mylog.debug("Initializing Particle Geometry >> >>>> >>>>> Handler.") >> >>>> >>>>> ---> 54 self._initialize_particle_handler() >> >>>> >>>>> 55 >> >>>> >>>>> 56 >> >>>> >>>>> >> >>>> >>>>> >> >>>> >>>>> >> >>>> >>>>> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/geometry/particle_geometry_handler.pyc >> >>>> >>>>> in _initialize_particle_handler(self) >> >>>> >>>>> 87 pf.domain_left_edge, >> >>>> >>>>> pf.domain_right_edge, >> >>>> >>>>> 88 [N, N, N], len(self.data_files)) >> >>>> >>>>> ---> 89 self._initialize_indices() >> >>>> >>>>> 90 self.oct_handler.finalize() >> >>>> >>>>> 91 self.max_level = self.oct_handler.max_level >> >>>> >>>>> >> >>>> >>>>> >> >>>> >>>>> >> >>>> >>>>> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/geometry/particle_geometry_handler.pyc >> >>>> >>>>> in _initialize_indices(self) >> >>>> >>>>> 109 npart = >> >>>> >>>>> sum(data_file.total_particles.values()) >> >>>> >>>>> 110 morton[ind:ind + npart] = \ >> >>>> >>>>> --> 111 self.io._initialize_index(data_file, >> >>>> >>>>> self.regions) >> >>>> >>>>> 112 ind += npart >> >>>> >>>>> 113 morton.sort() >> >>>> >>>>> >> >>>> >>>>> >> >>>> >>>>> >> >>>> >>>>> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/frontends/stream/io.pyc in >> >>>> >>>>> _initialize_index(self, data_file, regions) >> >>>> >>>>> 144 raise YTDomainOverflow(pos.min(axis=0), >> >>>> >>>>> pos.max(axis=0), >> >>>> >>>>> 145 >> >>>> >>>>> data_file.pf.domain_left_edge, >> >>>> >>>>> --> 146 >> >>>> >>>>> data_file.pf.domain_right_edge) >> >>>> >>>>> 147 regions.add_data_file(pos, >> >>>> >>>>> data_file.file_id) >> >>>> >>>>> 148 morton.append(compute_morton( >> >>>> >>>>> >> >>>> >>>>> YTDomainOverflow: Particle bounds [ 0. 0. 0.] and [ >> >>>> >>>>> 99.99999237 >> >>>> >>>>> 99.99999237 99.99999237] exceed domain bounds [ 0. 0. 0.] >> >>>> >>>>> code_length and >> >>>> >>>>> [ 1. 1. 1.] code_length >> >>>> >>>>> >> >>>> >>>>> >> >>>> >>>>> >> >>>> >>>>> On Thu, Jun 5, 2014 at 8:22 PM, Nathan Goldbaum >> >>>> >>>>> <nathan12343@gmail.com> wrote: >> >>>> >>>>>> >> >>>> >>>>>> Here's a worked out example that does what you're looking for >> >>>> >>>>>> using a fake 1 million particle dataset: >> >>>> >>>>>> >> >>>> >>>>>> >> >>>> >>>>>> http://nbviewer.ipython.org/gist/ngoldbaum/546d37869aafe71cfe38 >> >>>> >>>>>> >> >>>> >>>>>> In this notebook I make use of two key yt features: >> >>>> >>>>>> `load_particles`, and `covering_grid`. >> >>>> >>>>>> >> >>>> >>>>>> load_particles creates a "stream" dataset based on in-memory >> >>>> >>>>>> data >> >>>> >>>>>> fed in as a numpy array. This dataset acts just like an >> >>>> >>>>>> on-disk simulation >> >>>> >>>>>> dataset, but doesn't come with the baggage of needing to write >> >>>> >>>>>> a custom >> >>>> >>>>>> frontend to read a specific data format off disk. >> >>>> >>>>>> >> >>>> >>>>>> covering_grid is a way to generate uniform resolution data >> >>>> >>>>>> from >> >>>> >>>>>> an AMR dataset. It acts like a python dictionary where the >> >>>> >>>>>> keys are field >> >>>> >>>>>> names and returns 3D numpy arrays of whatever uniform >> >>>> >>>>>> resolution you specify >> >>>> >>>>>> when you create the covering_grid. >> >>>> >>>>>> >> >>>> >>>>>> Note that if you're using load_particles all of your data >> >>>> >>>>>> needs >> >>>> >>>>>> to live in memory. If your data is too big for that you'll >> >>>> >>>>>> need to write a >> >>>> >>>>>> frontend for your data format or use a memmap to an on-disk >> >>>> >>>>>> file somehow. >> >>>> >>>>>> I'm not an expert on that but others on the list should be >> >>>> >>>>>> able to help out. >> >>>> >>>>>> >> >>>> >>>>>> Hope that gets you well on your way :) >> >>>> >>>>>> >> >>>> >>>>>> -Nathan >> >>>> >>>>>> >> >>>> >>>>>> >> >>>> >>>>>> On Thu, Jun 5, 2014 at 5:04 PM, Desika Narayanan >> >>>> >>>>>> <dnarayan@haverford.edu> wrote: >> >>>> >>>>>>> >> >>>> >>>>>>> Hey Brendan, >> >>>> >>>>>>> >> >>>> >>>>>>> A couple of extra tools you might find helpful in conjunction >> >>>> >>>>>>> with Nathan's example of depositing the particles onto an >> >>>> >>>>>>> octree are at: >> >>>> >>>>>>> >> >>>> >>>>>>> http://paste.yt-project.org/show/4737/ >> >>>> >>>>>>> >> >>>> >>>>>>> Where I load a gadget snapshot, and then recover the >> >>>> >>>>>>> coordinates >> >>>> >>>>>>> and width of each cell. >> >>>> >>>>>>> >> >>>> >>>>>>> In response to your last question - the particles are >> >>>> >>>>>>> deposited >> >>>> >>>>>>> into an octree grid (so, you'll see that the cell sizes >> >>>> >>>>>>> aren't all the same >> >>>> >>>>>>> size). I don't know if depositing onto a regular NxNxN mesh >> >>>> >>>>>>> is possible, >> >>>> >>>>>>> though would be interested to hear if so. >> >>>> >>>>>>> >> >>>> >>>>>>> -d >> >>>> >>>>>>> >> >>>> >>>>>>> >> >>>> >>>>>>> On Thu, Jun 5, 2014 at 7:58 PM, Brendan Griffen >> >>>> >>>>>>> <brendan.f.griffen@gmail.com> wrote: >> >>>> >>>>>>>> >> >>>> >>>>>>>> Thanks. I'll get the "bleeding edge" version first then try >> >>>> >>>>>>>> your suggestions. Though I want to return the NxNxN array >> >>>> >>>>>>>> and be able to >> >>>> >>>>>>>> write this mesh to a file. It is *only* using the cic part >> >>>> >>>>>>>> of yt and it >> >>>> >>>>>>>> should return the mesh to be written? Just wanted to >> >>>> >>>>>>>> clarify? >> >>>> >>>>>>>> >> >>>> >>>>>>>> Thanks. >> >>>> >>>>>>>> Brendan >> >>>> >>>>>>>> >> >>>> >>>>>>>> >> >>>> >>>>>>>> On Thu, Jun 5, 2014 at 6:49 PM, Nathan Goldbaum >> >>>> >>>>>>>> <nathan12343@gmail.com> wrote: >> >>>> >>>>>>>>> >> >>>> >>>>>>>>> >> >>>> >>>>>>>>> >> >>>> >>>>>>>>> >> >>>> >>>>>>>>> On Thu, Jun 5, 2014 at 3:36 PM, John ZuHone >> >>>> >>>>>>>>> <jzuhone@gmail.com> wrote: >> >>>> >>>>>>>>>> >> >>>> >>>>>>>>>> Hi Brendan, >> >>>> >>>>>>>>>> >> >>>> >>>>>>>>>> Which version of yt are you using? >> >>>> >>>>>>>>>> >> >>>> >>>>>>>>>> If you're using 3.0, this is actually fairly easy. If you >> >>>> >>>>>>>>>> look in yt.fields.particle_fields.py, around line 85, you >> >>>> >>>>>>>>>> can see how this >> >>>> >>>>>>>>>> is done for the "particle_density" and "particle_mass" >> >>>> >>>>>>>>>> fields. Basically you >> >>>> >>>>>>>>>> can call a "deposit" method which takes the particle field >> >>>> >>>>>>>>>> quantity you want >> >>>> >>>>>>>>>> deposited and deposits it into cells. The underlying >> >>>> >>>>>>>>>> calculation is done >> >>>> >>>>>>>>>> using Cython, so it's fast. >> >>>> >>>>>>>>> >> >>>> >>>>>>>>> >> >>>> >>>>>>>>> And you shouldn't ever actually need to call these >> >>>> >>>>>>>>> "deposit" >> >>>> >>>>>>>>> functions, since "deposit" is exposed as a field type for >> >>>> >>>>>>>>> all datasets that >> >>>> >>>>>>>>> contain particles. >> >>>> >>>>>>>>> >> >>>> >>>>>>>>> Here is a notebook that does this for Enzo AMR data: >> >>>> >>>>>>>>> >> >>>> >>>>>>>>> >> >>>> >>>>>>>>> >> >>>> >>>>>>>>> http://nbviewer.ipython.org/gist/ngoldbaum/5e19e4e6cc2bf330149c >> >>>> >>>>>>>>> >> >>>> >>>>>>>>> This dataset contains about a million particles and >> >>>> >>>>>>>>> generates >> >>>> >>>>>>>>> a CIC deposition for the whole domain in about 6 seconds >> >>>> >>>>>>>>> from a cold start. >> >>>> >>>>>>>>> >> >>>> >>>>>>>>>> >> >>>> >>>>>>>>>> >> >>>> >>>>>>>>>> If you're using 2.x, then you can do the same thing, but >> >>>> >>>>>>>>>> it's >> >>>> >>>>>>>>>> not as straightforward. You can see how this works in >> >>>> >>>>>>>>>> yt.data_objects.universal_fields.py, around line 986, >> >>>> >>>>>>>>>> where the >> >>>> >>>>>>>>>> "particle_density" field is defined. Basically, it calls >> >>>> >>>>>>>>>> CICDeposit_3, which >> >>>> >>>>>>>>>> is in yt.utilities.lib.CICDeposit.pyx. >> >>>> >>>>>>>>>> >> >>>> >>>>>>>>>> Let me know if you need any more clarification. >> >>>> >>>>>>>>>> >> >>>> >>>>>>>>>> Best, >> >>>> >>>>>>>>>> >> >>>> >>>>>>>>>> John Z >> >>>> >>>>>>>>>> >> >>>> >>>>>>>>>> On Jun 5, 2014, at 6:07 PM, Brendan Griffen >> >>>> >>>>>>>>>> <brendan.f.griffen@gmail.com> wrote: >> >>>> >>>>>>>>>> >> >>>> >>>>>>>>>> > Hi, >> >>>> >>>>>>>>>> > >> >>>> >>>>>>>>>> > I was wondering if there were any Cython routines within >> >>>> >>>>>>>>>> > yt >> >>>> >>>>>>>>>> > which takes particle data and converts it into a >> >>>> >>>>>>>>>> > cloud-in-cell based mesh >> >>>> >>>>>>>>>> > which can be written to a file of my choosing. >> >>>> >>>>>>>>> >> >>>> >>>>>>>>> >> >>>> >>>>>>>>> What sort of mesh were you looking for? yt will internally >> >>>> >>>>>>>>> construct an octree if it is fed particle data. I'm not >> >>>> >>>>>>>>> sure whether this >> >>>> >>>>>>>>> octree can be saved to disk for later analysis. >> >>>> >>>>>>>>> >> >>>> >>>>>>>>> It's also possible to create a uniform resolution covering >> >>>> >>>>>>>>> grid containing field data for a deposited quantity, which >> >>>> >>>>>>>>> can be quite >> >>>> >>>>>>>>> easily saved to disk in a number of ways. >> >>>> >>>>>>>>> >> >>>> >>>>>>>>>> >> >>>> >>>>>>>>>> I heard a while ago there was some such functionality but >> >>>> >>>>>>>>>> it >> >>>> >>>>>>>>>> could be too far down the yt rabbit hole to be used as a >> >>>> >>>>>>>>>> standalone? Is this >> >>>> >>>>>>>>>> true? I have my own Python code for doing it but it just >> >>>> >>>>>>>>>> isn't fast enough >> >>>> >>>>>>>>>> and thought I'd ask the yt community if there were any >> >>>> >>>>>>>>>> wrapper tools >> >>>> >>>>>>>>>> available to boost the speed. >> >>>> >>>>>>>>>> > >> >>>> >>>>>>>>>> > Thanks. >> >>>> >>>>>>>>>> > Brendan >> >>>> >>>>>>>>>> > _______________________________________________ >> >>>> >>>>>>>>>> > yt-users mailing list >> >>>> >>>>>>>>>> > yt-users@lists.spacepope.org >> >>>> >>>>>>>>>> > >> >>>> >>>>>>>>>> > >> >>>> >>>>>>>>>> > http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >> >>>> >>>>>>>>>> >> >>>> >>>>>>>>>> _______________________________________________ >> >>>> >>>>>>>>>> yt-users mailing list >> >>>> >>>>>>>>>> yt-users@lists.spacepope.org >> >>>> >>>>>>>>>> >> >>>> >>>>>>>>>> >> >>>> >>>>>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >> >>>> >>>>>>>>> >> >>>> >>>>>>>>> >> >>>> >>>>>>>>> >> >>>> >>>>>>>>> _______________________________________________ >> >>>> >>>>>>>>> yt-users mailing list >> >>>> >>>>>>>>> yt-users@lists.spacepope.org >> >>>> >>>>>>>>> >> >>>> >>>>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >> >>>> >>>>>>>>> >> >>>> >>>>>>>> >> >>>> >>>>>>>> >> >>>> >>>>>>>> _______________________________________________ >> >>>> >>>>>>>> yt-users mailing list >> >>>> >>>>>>>> yt-users@lists.spacepope.org >> >>>> >>>>>>>> >> >>>> >>>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >> >>>> >>>>>>>> >> >>>> >>>>>>> >> >>>> >>>>>>> >> >>>> >>>>>>> _______________________________________________ >> >>>> >>>>>>> yt-users mailing list >> >>>> >>>>>>> yt-users@lists.spacepope.org >> >>>> >>>>>>> >> >>>> >>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >> >>>> >>>>>>> >> >>>> >>>>>> >> >>>> >>>>>> >> >>>> >>>>>> _______________________________________________ >> >>>> >>>>>> yt-users mailing list >> >>>> >>>>>> yt-users@lists.spacepope.org >> >>>> >>>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >> >>>> >>>>>> >> >>>> >>>>> >> >>>> >>>>> >> >>>> >>>>> _______________________________________________ >> >>>> >>>>> yt-users mailing list >> >>>> >>>>> yt-users@lists.spacepope.org >> >>>> >>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >> >>>> >>>>> >> >>>> >>>> >> >>>> >>>> >> >>>> >>>> _______________________________________________ >> >>>> >>>> yt-users mailing list >> >>>> >>>> yt-users@lists.spacepope.org >> >>>> >>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >> >>>> >>>> >> >>>> >>> >> >>>> >>> >> >>>> >>> _______________________________________________ >> >>>> >>> yt-users mailing list >> >>>> >>> yt-users@lists.spacepope.org >> >>>> >>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >> >>>> >>> >> >>>> >> >> >>>> >> >> >>>> >> _______________________________________________ >> >>>> >> yt-users mailing list >> >>>> >> yt-users@lists.spacepope.org >> >>>> >> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >> >>>> >> >> >>>> > >> >>>> > >> >>>> > _______________________________________________ >> >>>> > yt-users mailing list >> >>>> > yt-users@lists.spacepope.org >> >>>> > http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >> >>>> > >> >>>> >> >>>> >> >>>> _______________________________________________ >> >>>> yt-users mailing list >> >>>> yt-users@lists.spacepope.org >> >>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >> >>>> >> >>> >> >> >> >> >> >> _______________________________________________ >> >> yt-users mailing list >> >> yt-users@lists.spacepope.org >> >> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >> >> >> > >> > >> > _______________________________________________ >> > yt-users mailing list >> > yt-users@lists.spacepope.org >> > http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >> > >> _______________________________________________ >> yt-users mailing list >> yt-users@lists.spacepope.org >> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org > > > > _______________________________________________ > yt-users mailing list > yt-users@lists.spacepope.org > http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org > _______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
Hi Brendan, I think I've found the problem -- volume is making too many copies. I'm working on a fix. On Mon, Jun 9, 2014 at 12:56 PM, Nathan Goldbaum <nathan12343@gmail.com> wrote:
Yes, Matt and I are chatting off-list about this. I'll have more to say later hopefully.
On Mon, Jun 9, 2014 at 10:48 AM, Brendan Griffen <brendan.f.griffen@gmail.com> wrote:
Hi Nathan, I'm not sure if you missed it but earlier Matt asked if there was some copying which shouldn't be happening?
Brendan
On Mon, Jun 9, 2014 at 1:22 PM, Nathan Goldbaum <nathan12343@gmail.com> wrote:
On Mon, Jun 9, 2014 at 12:32 AM, Brendan Griffen <brendan.f.griffen@gmail.com> wrote:
Thanks Nathan. Comparing the two though:
26 1536.664 MiB 1130.480 MiB uniform_array = grid_object['deposit', 'all_cic']
24 1285.703 MiB 890.539 MiB cic_density = ad["deposit", "all_cic"]
Isn't it the uniform grid which uses the most memory? I'm running the profiling now and will get back to you. I hope in spite of the fact that it does crash it will still give me some useful output.
That's true - I was thinking it might be more memory efficient since there is no need to construct the covering grid in addition to the octree.
Brendan
On Mon, Jun 9, 2014 at 2:05 AM, Nathan Goldbaum <nathan12343@gmail.com> wrote:
Hey Brendan,
Could you try running your script using the memory_profiler module on pypi?
Here's an example script that uses the memory profiler: http://paste.yt-project.org/show/4748/
and the output for that script: http://paste.yt-project.org/show/4749/
For what it's worth, it does indeed look like matt's suggestion to use load_uniform_grid is an option, and might be more memory efficient in the end since you will go directly to the uniform grid you want without creating an octree. Here's an example: http://paste.yt-project.org/show/4750/
Here's the memory usage information for that example: http://paste.yt-project.org/show/4751/
I used a 256^3 uniform grid with normally distributed random data - I'm not sure whether it will also be more memory efficient in your case.
On Sun, Jun 8, 2014 at 9:32 PM, Brendan Griffen <brendan.f.griffen@gmail.com> wrote:
This is the full error if it helps at all? It is indeed, loading in all of the quantities.
Loading particles... --> Loading particle type: 1 yt : [INFO ] 2014-06-08 15:12:05,540 Parameters: current_time = 0.0 yt : [INFO ] 2014-06-08 15:12:05,540 Parameters: domain_dimensions = [2 2 2] yt : [INFO ] 2014-06-08 15:12:05,541 Parameters: domain_left_edge = [ 0. 0. 0.] yt : [INFO ] 2014-06-08 15:12:05,542 Parameters: domain_right_edge = [ 100. 100. 100.] yt : [INFO ] 2014-06-08 15:12:05,542 Parameters: cosmological_simulation = 0.0 yt : [INFO ] 2014-06-08 15:12:05,548 Allocating for 1.074e+09 particles yt : [INFO ] 2014-06-08 15:16:16,195 Identified 7.584e+07 octs yt : [INFO ] 2014-06-08 15:16:16,299 Loading field plugins. yt : [INFO ] 2014-06-08 15:16:16,299 Loaded angular_momentum (8 new fields) yt : [INFO ] 2014-06-08 15:16:16,299 Loaded astro (14 new fields) yt : [INFO ] 2014-06-08 15:16:16,300 Loaded cosmology (20 new fields) yt : [INFO ] 2014-06-08 15:16:16,300 Loaded fluid (56 new fields) yt : [INFO ] 2014-06-08 15:16:16,301 Loaded fluid_vector (88 new fields) yt : [INFO ] 2014-06-08 15:16:16,301 Loaded geometric (103 new fields) yt : [INFO ] 2014-06-08 15:16:16,301 Loaded local (103 new fields) yt : [INFO ] 2014-06-08 15:16:16,302 Loaded magnetic_field (109 new fields) yt : [INFO ] 2014-06-08 15:16:16,302 Loaded species (109 new fields)
--------------------------------------------------------------------------- MemoryError Traceback (most recent call last)
/nfs/blank/h4231/bgriffen/data/lib/yt-x86_64/lib/python2.7/site-packages/IPython/utils/py3compat.pyc in execfile(fname, *where) 202 else: 203 filename = fname --> 204 __builtin__.execfile(filename, *where)
/nfs/blank/h4231/bgriffen/work/projects/caterpillar/c2ray/cic/ytcic.py in <module>() 99 slc.set_figure_size(4) 100 slc.save() --> 101 102 for ndim in dimlist: 103 print
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc in __getitem__(self, key) 218 return self.field_data[f] 219 else: --> 220 self.get_data(f) 221 # fi.units is the unit expression string. We depend on the registry 222 # hanging off the dataset to define this unit object.
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc in get_data(self, fields) 627 628 fields_to_generate += gen_fluids + gen_particles --> 629 self._generate_fields(fields_to_generate) 630 631 def _generate_fields(self, fields_to_generate):
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc in _generate_fields(self, fields_to_generate) 644 fi = self.pf._get_field_info(*field) 645 try: --> 646 fd = self._generate_field(field) 647 if type(fd) == np.ndarray: 648 fd = self.pf.arr(fd, fi.units)
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc in _generate_field(self, field) 255 tr = self._generate_particle_field(field) 256 else: --> 257 tr = self._generate_fluid_field(field) 258 if tr is None: 259 raise YTCouldNotGenerateField(field, self.pf)
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc in _generate_fluid_field(self, field) 273 finfo.check_available(gen_obj) 274 except NeedsGridType as ngt_exception: --> 275 rv = self._generate_spatial_fluid(field, ngt_exception.ghost_zones) 276 else: 277 rv = finfo(gen_obj)
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc in _generate_spatial_fluid(self, field, ngz) 289 o = self._current_chunk.objs[0] 290 with o._activate_cache(): --> 291 ind += o.select(self.selector, self[field], rv, ind) 292 else: 293 chunks = self.index._chunk(self, "spatial", ngz = ngz)
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc in __getitem__(self, key) 218 return self.field_data[f] 219 else: --> 220 self.get_data(f) 221 # fi.units is the unit expression string. We depend on the registry 222 # hanging off the dataset to define this unit object.
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc in get_data(self, fields) 627 628 fields_to_generate += gen_fluids + gen_particles --> 629 self._generate_fields(fields_to_generate) 630 631 def _generate_fields(self, fields_to_generate):
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc in _generate_fields(self, fields_to_generate) 644 fi = self.pf._get_field_info(*field) 645 try: --> 646 fd = self._generate_field(field) 647 if type(fd) == np.ndarray: 648 fd = self.pf.arr(fd, fi.units)
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc in _generate_field(self, field) 255 tr = self._generate_particle_field(field) 256 else: --> 257 tr = self._generate_fluid_field(field) 258 if tr is None: 259 raise YTCouldNotGenerateField(field, self.pf)
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc in _generate_fluid_field(self, field) 275 rv = self._generate_spatial_fluid(field, ngt_exception.ghost_zones) 276 else: --> 277 rv = finfo(gen_obj) 278 return rv 279
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/fields/derived_field.pyc in __call__(self, data) 176 "for %s" % (self.name,)) 177 with self.unit_registry(data): --> 178 dd = self._function(self, data) 179 for field_name in data.keys(): 180 if field_name not in original_fields:
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/fields/particle_fields.pyc in particle_cic(field, data) 113 def particle_cic(field, data): 114 pos = data[ptype, coord_name] --> 115 d = data.deposit(pos, [data[ptype, mass_name]], method = "cic") 116 d = data.apply_units(d, data[ptype, mass_name].units) 117 d /= data["index", "cell_volume"]
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/octree_subset.pyc in deposit(self, positions, fields, method) 167 mylog.debug("Depositing %s (%s^3) particles into %s Octs", 168 positions.shape[0], positions.shape[0]**0.3333333, nvals[-1]) --> 169 pos = np.asarray(positions.convert_to_units("code_length"), 170 dtype="float64") 171 # We should not need the following if we know in advance all our fields
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/units/yt_array.pyc in convert_to_units(self, units) 366 367 self.units = new_units --> 368 self *= conversion_factor 369 return self 370
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/units/yt_array.pyc in __imul__(self, other) 667 """ See __mul__. """ 668 oth = sanitize_units_mul(self, other) --> 669 return np.multiply(self, oth, out=self) 670 671 def __div__(self, right_object):
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/units/yt_array.pyc in __array_wrap__(self, out_arr, context) 966 # casting to YTArray avoids creating a YTQuantity with size > 1 967 return YTArray(np.array(out_arr, unit)) --> 968 return ret_class(np.array(out_arr), unit) 969 970
MemoryError:
On Sun, Jun 8, 2014 at 10:50 PM, Matthew Turk <matthewturk@gmail.com> wrote: > > Hi Brendan, > > On Sun, Jun 8, 2014 at 9:21 PM, Brendan Griffen > <brendan.f.griffen@gmail.com> wrote: > > Hi Matt, > > > > Thanks for your detailed email. Forgive my naivety but why do you > > need the > > oct-tree in the first place? I have a my own fortran code for > > constructing a > > cloud in cell mesh and it uses very little overhead (just the n^3 > > grid and > > the particle data itself). I then calculate the dx,dy,dzs to the > > nearest 8 > > grid points and distribute accordingly in a omp loop which is done > > in a > > fraction of a second. Does the situation with yt come about (oct > > tree etc.) > > necessarily because of the way it handles particle data? Is it > > essentially > > used to map the particles to domains in the grid or something? > > That's not naive at all. There are two reasons -- > > 1) The octree is used for indexing for neighbor lookups and > early-termination of region selection for particles > 2) The octree is used to estimate the "required resolution" for any > operation that requires a space-filling value. (i.e., any time that > a > particle becomes a volume.) > > Projections in yt are adaptive, in that they project down to the > finest appropriate resolution. There's also the "arbitrary_grid" > operation, which does precisely what you're describing, but as it > stands right now the octree gets constructed at time of instantiation > of the indexing system. Thinking it over, you may be able to avoid > that completely by not using load_particles and instead using > load_uniform_grid and supplying your desired dimensions. The field > names should be the same. > > > > > The machine I max memory on has 128GB and the snapshots are using > > 1024^3 > > particles. Do you have any idea of how much memory the oct-tree > > uses as a > > function of particle/grid number? I am going to try on a 256GB > > machine > > (though this is a bit of a hassle). I'll see how I go. > > I am disappointed that it's blowing out your RAM. This week I will > try to get some memory profiling done. Could you file a bug to this > effect, which will help me track it? Peak memory usage during > indexing should only be 64 bits * Nparticles, unless you're using > load_particles, in which case all the fields will *also* have to be > in > memory. It's about 8 gigabytes per field. So, I think there's > something going wrong. > > > > > Thanks. > > > > Brendan > > > > > > On Sun, Jun 8, 2014 at 6:25 PM, Matthew Turk > > <matthewturk@gmail.com> wrote: > >> > >> Hi all, > >> > >> I feel like I owe a brief explanation of why things are tricky > >> right > >> now, what we're planning on doing, and how we're experimenting and > >> developing. > >> > >> Presently, the particle geometry handlers build a single mesh from > >> all > >> particles in the dataset, along with a coarse bitmask that > >> correlates > >> files to regions in the domain. This requires the allocation of a > >> single int64 array of size Nparticles, which is sorted in place > >> and > >> then fed into an octree construction algorithm that then spits > >> back > >> out the mesh. Each octree component contains 3 64-bit integers > >> and > >> eitehr a void pointer or a pointer to eight other octs. > >> Increasing > >> n_ref decreases the number of octs in this mesh; when smoothing > >> operaitons are conducted, a second "index" mesh is created for > >> looking > >> up particles near mesh points. Mesh points are used for adaptive > >> resolution smoothing and other "deposit particles on the grid > >> somehow" > >> operations (including SPH kernel). > >> > >> Anyway, because right now it requires a global mesh to be > >> constructed, > >> this is expensive and requires holding a 64-bit integer in memory > >> for > >> each particle. I think if you're loading the particles in > >> differently > >> there is some additional overhead as well, but I'm still a bit > >> surprised you OOM on a 1024^3 dataset. > >> > >> In general, we don't *need* this global mesh; is can be > >> constructed as > >> required, which would speed up both the initial index phase as > >> well as > >> the final meshing process. I got about 50% of the way to > >> implementing > >> this last fall, but because of various concerns and deadlines I > >> haven't finished it. I intend to get back to it probably in July, > >> right after we put out a 3.0, so that we can have it in time for > >> 3.1. > >> In principle this will make the particle codes much more similar > >> to > >> ARTIO, in that the mesh will be constructed only as required and > >> discarded when no longer required, which will make them much more > >> memory efficient. > >> > >> But, getting a single mesh for extremely large data is a very high > >> priority; right now for the 10240^3 run we've been loading up > >> individual sub-chunks, which I want to stop doing. > >> > >> From the technical perspective, these are the things that need to > >> happen on the yt side for particle datasets to move to this "lazy" > >> mode of loading; most of this is based on things learned from 2HOT > >> and > >> ARTIO, and will involve converting to a forest-of-octrees. > >> > >> * Split into spatially-organized subchunks of > >> ParticleOctreeSubset > >> objects, such that these map 1:Nfiles, and that can be constructed > >> on > >> the fly. > >> * Construct a dual-mesh of the bitmask "ParticleRegion" object > >> that > >> will help with identifying neighbors to a given oct cell, so that > >> if > >> we're inside one octree we know which neighbor octrees to grab if > >> we > >> need particles for smoothing things (fast boundary particle > >> identification is later down the road) > >> * Parallel sort of particles, or using the parallel ring > >> function; > >> may not be necessary after all > >> > >> All of this is doable, and I'd be happy to work with people if > >> they'd > >> like to take a shot at implementing it, but I've mostly put it on > >> my > >> list for post-3.0. > >> > >> -Matt > >> > >> On Sun, Jun 8, 2014 at 2:43 PM, Nathan Goldbaum > >> <nathan12343@gmail.com> > >> wrote: > >> > > >> > > >> > > >> > On Sun, Jun 8, 2014 at 12:27 PM, Brendan Griffen > >> > <brendan.f.griffen@gmail.com> wrote: > >> >> > >> >> Also, how do I construct just a zero filled yt array with > >> >> dimensions > >> >> (ndim,ndim,ndim)? Thanks > >> > > >> > > >> > > >> > from yt import YTArray > >> > from numpy import np > >> > > >> > arr = YTArray(np.zeros([ndim, ndim, ndim]), > >> > input_units=units_string) > >> > > >> > or alternatively: > >> > > >> > from yt.units import kiloparsec > >> > > >> > arr = kiloparsec*np.zeros([ndim, ndim, ndim]) > >> > > >> > it doesn't have to be kiloparsec - you can compose the units you > >> > want > >> > out of > >> > any of the unit symbols that live in yt.units. > >> > > >> > See this page for a ton more detail about yt's new unit system: > >> > http://yt-project.org/docs/dev-3.0/analyzing/units/index.html > >> > > >> >> > >> >> > >> >> Brendan > >> >> > >> >> > >> >> On Sun, Jun 8, 2014 at 3:26 PM, Brendan Griffen > >> >> <brendan.f.griffen@gmail.com> wrote: > >> >>> > >> >>> Hi, > >> >>> > >> >>> Since I get memory errors. Could I not just read in the blocks > >> >>> of the > >> >>> output individually then basically stack the mesh each time. > >> >>> That way > >> >>> not > >> >>> every single particle of the snapshot has to be loaded at the > >> >>> same > >> >>> time. > >> >>> Would that just be a case of doing > >> >>> > >> >>> level = int(math.log(ndim,2)) > >> >>> cg = ds.covering_grid(level=level, > >> >>> left_edge=[0,0,0],dims=[ndim,ndim,ndim]) > >> >>> arr = cg['deposit', 'all_density'] > >> >>> arrall += arr > >> >>> > >> >>> in a loop over each HDF5 block? > >> > > >> > > >> > It's likely that the memory use is dominated by the octree > >> > rather than > >> > the > >> > covering grid. This is with 1024^3 particles, correct? > >> > > >> > You can probably significantly reduce the memory used by the > >> > octree by > >> > increasing n_ref in the call to load_particles. > >> > > >> > See this page for more detail about load_particles: > >> > > >> > > >> > http://yt-project.org/docs/dev-3.0/examining/loading_data.html#generic-parti... > >> > > >> > Larger n_ref means fewer octree cells (lower resolution), but it > >> > also > >> > means > >> > lower poisson noise and lower memory use. > >> > > >> > Alternatively, as Matt suggested, you could break your 1024^3 > >> > ensemble > >> > of > >> > particles up into chunks, loop over the chunk, creating a > >> > particle > >> > octree > >> > and then a covering grid for each subset of the particles. Your > >> > final > >> > covering grid is just the sub of the covering grids for each > >> > subset of > >> > particles. > >> > > >> >>> > >> >>> > >> >>> Thanks. > >> >>> Brendan > >> >>> > >> >>> > >> >>> > >> >>> > >> >>> On Fri, Jun 6, 2014 at 7:26 PM, Matthew Turk > >> >>> <matthewturk@gmail.com> > >> >>> wrote: > >> >>>> > >> >>>> > >> >>>> On Jun 6, 2014 4:54 PM, "Brendan Griffen" > >> >>>> <brendan.f.griffen@gmail.com> > >> >>>> wrote: > >> >>>> > > >> >>>> > OK great. It is very low resolution but it worked. Thanks > >> >>>> > for all > >> >>>> > your > >> >>>> > help. My higher resolution run 1024^3 in 100 Mpc seems to > >> >>>> > crash on > >> >>>> > 128GB > >> >>>> > memory machine. I might have to look elsewhere. > >> >>>> > > >> >>>> > >> >>>> If you are looking for low resolution extraction you can > >> >>>> tune the > >> >>>> memory usage by changing the parameter n_ref to something > >> >>>> higher. > >> >>>> > >> >>>> Supporting extremely large datasets in a single mesh is on > >> >>>> the > >> >>>> roadmap > >> >>>> for the late summer or fall, after a 3.0 release goes out. > >> >>>> For now > >> >>>> you can > >> >>>> also extract before you load in; this is sort of how we are > >> >>>> supporting an > >> >>>> INCITE project with very large particle counts. > >> >>>> > >> >>>> > >> >>>> > Also, I normally use Canopy distribution but I just use an > >> >>>> > alias to > >> >>>> > loadyt which erases my PYTHONPATH and I can't access scipy > >> >>>> > and a > >> >>>> > few other > >> >>>> > libraries any more. What is the best practice here? Should > >> >>>> > I just > >> >>>> > manually > >> >>>> > export PYTHONPATH and point to the libraries need in canopy > >> >>>> > or can > >> >>>> > they play > >> >>>> > nice together? > >> >>>> > > >> >>>> > Thanks. > >> >>>> > > >> >>>> > BG > >> >>>> > > >> >>>> > > >> >>>> > On Fri, Jun 6, 2014 at 2:54 PM, Nathan Goldbaum > >> >>>> > <nathan12343@gmail.com> wrote: > >> >>>> >> > >> >>>> >> > >> >>>> >> > >> >>>> >> > >> >>>> >> On Fri, Jun 6, 2014 at 11:48 AM, Brendan Griffen > >> >>>> >> <brendan.f.griffen@gmail.com> wrote: > >> >>>> >>> > >> >>>> >>> OK great. Thanks. I just wanted a homogeneous mesh. 512^3 > >> >>>> >>> with no > >> >>>> >>> nesting of any kind. Though when I plot the image it > >> >>>> >>> looks like > >> >>>> >>> it is > >> >>>> >>> assigning particles incorrectly (low resolution on the > >> >>>> >>> outside). > >> >>>> >>> This is > >> >>>> >>> just a test image. > >> >>>> >>> > >> >>>> >> > >> >>>> >> The SlicePlot is visualizing the octree so there is less > >> >>>> >> resolution > >> >>>> >> where there are fewer particles. If you want to visualize > >> >>>> >> the > >> >>>> >> covering grid > >> >>>> >> you're going to need to visualize that separately. > >> >>>> >> > >> >>>> >>> > >> >>>> >>> ds = yt.load_particles(data, length_unit=3.08e24, > >> >>>> >>> mass_unit=1.9891e33,bbox=bbox) > >> >>>> >>> > >> >>>> >>> ad = ds.all_data() > >> >>>> >>> print ad['deposit', 'all_cic'] > >> >>>> >>> slc = yt.SlicePlot(ds, 2, ('deposit', 'all_cic')) > >> >>>> >>> slc.set_figure_size(4) > >> >>>> >>> cg = ds.covering_grid(level=9, > >> >>>> >>> left_edge=[0,0,0],dims=[512,512,512]) > >> >>>> >>> > >> >>>> >> > >> >>>> >> To actually produce the uniform resolution ndarray, you're > >> >>>> >> going > >> >>>> >> to > >> >>>> >> need to do something like: > >> >>>> >> > >> >>>> >> array = cg[('deposit', 'all_cic')] > >> >>>> >> > >> >>>> >> array will then be a 3D array you can do whatever you want > >> >>>> >> with. > >> >>>> >> By > >> >>>> >> default it has units, but to strip them off you'll just > >> >>>> >> need to > >> >>>> >> cast to > >> >>>> >> ndarray: > >> >>>> >> > >> >>>> >> array_without_units = array.v > >> >>>> >> > >> >>>> >> > >> >>>> >>> > >> >>>> >>> Also, is there a way to load multiple particle types? > >> >>>> >>> > >> >>>> >>> Do I just need to stack the particles into the array > >> >>>> >>> here? > >> >>>> >>> > >> >>>> >>> data = {'particle_position_x': pos[:,0], > >> >>>> >>> 'particle_position_y': pos[:,1], > >> >>>> >>> 'particle_position_z': pos[:,2], > >> >>>> >>> 'particle_mass': np.array([mpart]*npart)} > >> >>>> >>> > >> >>>> >>> Then feed it in as usual? > >> >>>> >> > >> >>>> >> > >> >>>> >> That's right, although if the particle masses are > >> >>>> >> different for > >> >>>> >> the > >> >>>> >> different particle types that code snippet will need to be > >> >>>> >> generalized to > >> >>>> >> handle that. > >> >>>> >> > >> >>>> >> I think in principle it should be possible to make > >> >>>> >> load_particles > >> >>>> >> handle different particle types just like an SPH dataset > >> >>>> >> that > >> >>>> >> contains > >> >>>> >> multiple particle types, but right now that hasn't been > >> >>>> >> implemented yet. > >> >>>> >> > >> >>>> >>> > >> >>>> >>> > >> >>>> >>> Brendan > >> >>>> >>> > >> >>>> >>> > >> >>>> >>> On Thu, Jun 5, 2014 at 9:44 PM, Nathan Goldbaum > >> >>>> >>> <nathan12343@gmail.com> wrote: > >> >>>> >>>> > >> >>>> >>>> That's right, you can set that via the bbox keyword > >> >>>> >>>> parameter > >> >>>> >>>> for > >> >>>> >>>> load_particles. I'd urge you to take a look at the > >> >>>> >>>> docstrings > >> >>>> >>>> and source > >> >>>> >>>> code for load_particles. > >> >>>> >>>> > >> >>>> >>>> > >> >>>> >>>> On Thu, Jun 5, 2014 at 6:34 PM, Brendan Griffen > >> >>>> >>>> <brendan.f.griffen@gmail.com> wrote: > >> >>>> >>>>> > >> >>>> >>>>> Thanks very much Nathan. I tried to load in my own data > >> >>>> >>>>> but I > >> >>>> >>>>> think there are too many particles or I have to > >> >>>> >>>>> specifically > >> >>>> >>>>> set the domain > >> >>>> >>>>> size. > >> >>>> >>>>> > >> >>>> >>>>> In this area: > >> >>>> >>>>> > >> >>>> >>>>> data = {'particle_position_x': pos[:,0], > >> >>>> >>>>> 'particle_position_y': pos[:,1], > >> >>>> >>>>> 'particle_position_z': pos[:,2], > >> >>>> >>>>> 'particle_mass': np.array([mpart]*npart)} > >> >>>> >>>>> > >> >>>> >>>>> ds = yt.load_particles(data, length_unit=3.08e24, > >> >>>> >>>>> mass_unit=1.9891e36) > >> >>>> >>>>> ad = ds.all_data() > >> >>>> >>>>> print ad['deposit', 'all_cic'] > >> >>>> >>>>> > >> >>>> >>>>> In [3]: run ytcic.py > >> >>>> >>>>> yt : [INFO ] 2014-06-05 21:29:06,183 Parameters: > >> >>>> >>>>> current_time > >> >>>> >>>>> = 0.0 > >> >>>> >>>>> yt : [INFO ] 2014-06-05 21:29:06,183 Parameters: > >> >>>> >>>>> domain_dimensions = [2 2 2] > >> >>>> >>>>> yt : [INFO ] 2014-06-05 21:29:06,184 Parameters: > >> >>>> >>>>> domain_left_edge = [ 0. 0. 0.] > >> >>>> >>>>> yt : [INFO ] 2014-06-05 21:29:06,185 Parameters: > >> >>>> >>>>> domain_right_edge = [ 1. 1. 1.] > >> >>>> >>>>> yt : [INFO ] 2014-06-05 21:29:06,185 Parameters: > >> >>>> >>>>> cosmological_simulation = 0.0 > >> >>>> >>>>> yt : [INFO ] 2014-06-05 21:29:06,188 Allocating for > >> >>>> >>>>> 1.342e+08 > >> >>>> >>>>> particles > >> >>>> >>>>> > >> >>>> >>>>> > >> >>>> >>>>> > >> >>>> >>>>> --------------------------------------------------------------------------- > >> >>>> >>>>> YTDomainOverflow Traceback > >> >>>> >>>>> (most > >> >>>> >>>>> recent > >> >>>> >>>>> call last) > >> >>>> >>>>> > >> >>>> >>>>> > >> >>>> >>>>> > >> >>>> >>>>> /nfs/blank/h4231/bgriffen/data/lib/yt-x86_64/lib/python2.7/site-packages/IPython/utils/py3compat.pyc > >> >>>> >>>>> in execfile(fname, *where) > >> >>>> >>>>> 202 else: > >> >>>> >>>>> 203 filename = fname > >> >>>> >>>>> --> 204 __builtin__.execfile(filename, > >> >>>> >>>>> *where) > >> >>>> >>>>> > >> >>>> >>>>> > >> >>>> >>>>> > >> >>>> >>>>> > >> >>>> >>>>> /nfs/blank/h4231/bgriffen/work/projects/caterpillar/c2ray/cic/ytcic.py in > >> >>>> >>>>> <module>() > >> >>>> >>>>> 52 > >> >>>> >>>>> 53 ad = ds.all_data() > >> >>>> >>>>> ---> 54 print ad['deposit', 'all_cic'] > >> >>>> >>>>> 55 slc = yt.SlicePlot(ds, 2, ('deposit', > >> >>>> >>>>> 'all_cic')) > >> >>>> >>>>> 56 slc.set_figure_size(4) > >> >>>> >>>>> > >> >>>> >>>>> > >> >>>> >>>>> > >> >>>> >>>>> > >> >>>> >>>>> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc > >> >>>> >>>>> in __getitem__(self, key) > >> >>>> >>>>> 205 Returns a single field. Will add if > >> >>>> >>>>> necessary. > >> >>>> >>>>> 206 """ > >> >>>> >>>>> --> 207 f = self._determine_fields([key])[0] > >> >>>> >>>>> 208 if f not in self.field_data and key not > >> >>>> >>>>> in > >> >>>> >>>>> self.field_data: > >> >>>> >>>>> 209 if f in self._container_fields: > >> >>>> >>>>> > >> >>>> >>>>> > >> >>>> >>>>> > >> >>>> >>>>> > >> >>>> >>>>> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc > >> >>>> >>>>> in _determine_fields(self, fields) > >> >>>> >>>>> 453 raise > >> >>>> >>>>> YTFieldNotParseable(field) > >> >>>> >>>>> 454 ftype, fname = field > >> >>>> >>>>> --> 455 finfo = > >> >>>> >>>>> self.pf._get_field_info(ftype, > >> >>>> >>>>> fname) > >> >>>> >>>>> 456 else: > >> >>>> >>>>> 457 fname = field > >> >>>> >>>>> > >> >>>> >>>>> > >> >>>> >>>>> > >> >>>> >>>>> > >> >>>> >>>>> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/static_output.pyc > >> >>>> >>>>> in _get_field_info(self, ftype, fname) > >> >>>> >>>>> 445 _last_finfo = None > >> >>>> >>>>> 446 def _get_field_info(self, ftype, fname = > >> >>>> >>>>> None): > >> >>>> >>>>> --> 447 self.index > >> >>>> >>>>> 448 if fname is None: > >> >>>> >>>>> 449 ftype, fname = "unknown", ftype > >> >>>> >>>>> > >> >>>> >>>>> > >> >>>> >>>>> > >> >>>> >>>>> > >> >>>> >>>>> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/static_output.pyc > >> >>>> >>>>> in index(self) > >> >>>> >>>>> 277 raise RuntimeError("You should > >> >>>> >>>>> not > >> >>>> >>>>> instantiate Dataset.") > >> >>>> >>>>> 278 self._instantiated_index = > >> >>>> >>>>> self._index_class( > >> >>>> >>>>> --> 279 self, > >> >>>> >>>>> dataset_type=self.dataset_type) > >> >>>> >>>>> 280 # Now we do things that we need an > >> >>>> >>>>> instantiated index for > >> >>>> >>>>> 281 # ...first off, we create our > >> >>>> >>>>> field_info > >> >>>> >>>>> now. > >> >>>> >>>>> > >> >>>> >>>>> > >> >>>> >>>>> > >> >>>> >>>>> > >> >>>> >>>>> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/frontends/stream/data_structures.pyc > >> >>>> >>>>> in __init__(self, pf, dataset_type) > >> >>>> >>>>> 942 def __init__(self, pf, dataset_type = > >> >>>> >>>>> None): > >> >>>> >>>>> 943 self.stream_handler = pf.stream_handler > >> >>>> >>>>> --> 944 super(StreamParticleIndex, > >> >>>> >>>>> self).__init__(pf, > >> >>>> >>>>> dataset_type) > >> >>>> >>>>> 945 > >> >>>> >>>>> 946 def _setup_data_io(self): > >> >>>> >>>>> > >> >>>> >>>>> > >> >>>> >>>>> > >> >>>> >>>>> > >> >>>> >>>>> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/geometry/particle_geometry_handler.pyc > >> >>>> >>>>> in __init__(self, pf, dataset_type) > >> >>>> >>>>> 48 self.directory = > >> >>>> >>>>> os.path.dirname(self.index_filename) > >> >>>> >>>>> 49 self.float_type = np.float64 > >> >>>> >>>>> ---> 50 super(ParticleIndex, self).__init__(pf, > >> >>>> >>>>> dataset_type) > >> >>>> >>>>> 51 > >> >>>> >>>>> 52 def _setup_geometry(self): > >> >>>> >>>>> > >> >>>> >>>>> > >> >>>> >>>>> > >> >>>> >>>>> > >> >>>> >>>>> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/geometry/geometry_handler.pyc > >> >>>> >>>>> in __init__(self, pf, dataset_type) > >> >>>> >>>>> 54 > >> >>>> >>>>> 55 mylog.debug("Setting up domain > >> >>>> >>>>> geometry.") > >> >>>> >>>>> ---> 56 self._setup_geometry() > >> >>>> >>>>> 57 > >> >>>> >>>>> 58 mylog.debug("Initializing data grid > >> >>>> >>>>> data IO") > >> >>>> >>>>> > >> >>>> >>>>> > >> >>>> >>>>> > >> >>>> >>>>> > >> >>>> >>>>> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/geometry/particle_geometry_handler.pyc > >> >>>> >>>>> in _setup_geometry(self) > >> >>>> >>>>> 52 def _setup_geometry(self): > >> >>>> >>>>> 53 mylog.debug("Initializing Particle > >> >>>> >>>>> Geometry > >> >>>> >>>>> Handler.") > >> >>>> >>>>> ---> 54 self._initialize_particle_handler() > >> >>>> >>>>> 55 > >> >>>> >>>>> 56 > >> >>>> >>>>> > >> >>>> >>>>> > >> >>>> >>>>> > >> >>>> >>>>> > >> >>>> >>>>> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/geometry/particle_geometry_handler.pyc > >> >>>> >>>>> in _initialize_particle_handler(self) > >> >>>> >>>>> 87 pf.domain_left_edge, > >> >>>> >>>>> pf.domain_right_edge, > >> >>>> >>>>> 88 [N, N, N], > >> >>>> >>>>> len(self.data_files)) > >> >>>> >>>>> ---> 89 self._initialize_indices() > >> >>>> >>>>> 90 self.oct_handler.finalize() > >> >>>> >>>>> 91 self.max_level = > >> >>>> >>>>> self.oct_handler.max_level > >> >>>> >>>>> > >> >>>> >>>>> > >> >>>> >>>>> > >> >>>> >>>>> > >> >>>> >>>>> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/geometry/particle_geometry_handler.pyc > >> >>>> >>>>> in _initialize_indices(self) > >> >>>> >>>>> 109 npart = > >> >>>> >>>>> sum(data_file.total_particles.values()) > >> >>>> >>>>> 110 morton[ind:ind + npart] = \ > >> >>>> >>>>> --> 111 > >> >>>> >>>>> self.io._initialize_index(data_file, > >> >>>> >>>>> self.regions) > >> >>>> >>>>> 112 ind += npart > >> >>>> >>>>> 113 morton.sort() > >> >>>> >>>>> > >> >>>> >>>>> > >> >>>> >>>>> > >> >>>> >>>>> > >> >>>> >>>>> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/frontends/stream/io.pyc in > >> >>>> >>>>> _initialize_index(self, data_file, regions) > >> >>>> >>>>> 144 raise > >> >>>> >>>>> YTDomainOverflow(pos.min(axis=0), > >> >>>> >>>>> pos.max(axis=0), > >> >>>> >>>>> 145 > >> >>>> >>>>> data_file.pf.domain_left_edge, > >> >>>> >>>>> --> 146 > >> >>>> >>>>> data_file.pf.domain_right_edge) > >> >>>> >>>>> 147 regions.add_data_file(pos, > >> >>>> >>>>> data_file.file_id) > >> >>>> >>>>> 148 morton.append(compute_morton( > >> >>>> >>>>> > >> >>>> >>>>> YTDomainOverflow: Particle bounds [ 0. 0. 0.] and [ > >> >>>> >>>>> 99.99999237 > >> >>>> >>>>> 99.99999237 99.99999237] exceed domain bounds [ 0. 0. > >> >>>> >>>>> 0.] > >> >>>> >>>>> code_length and > >> >>>> >>>>> [ 1. 1. 1.] code_length > >> >>>> >>>>> > >> >>>> >>>>> > >> >>>> >>>>> > >> >>>> >>>>> On Thu, Jun 5, 2014 at 8:22 PM, Nathan Goldbaum > >> >>>> >>>>> <nathan12343@gmail.com> wrote: > >> >>>> >>>>>> > >> >>>> >>>>>> Here's a worked out example that does what you're > >> >>>> >>>>>> looking for > >> >>>> >>>>>> using a fake 1 million particle dataset: > >> >>>> >>>>>> > >> >>>> >>>>>> > >> >>>> >>>>>> > >> >>>> >>>>>> http://nbviewer.ipython.org/gist/ngoldbaum/546d37869aafe71cfe38 > >> >>>> >>>>>> > >> >>>> >>>>>> In this notebook I make use of two key yt features: > >> >>>> >>>>>> `load_particles`, and `covering_grid`. > >> >>>> >>>>>> > >> >>>> >>>>>> load_particles creates a "stream" dataset based on > >> >>>> >>>>>> in-memory > >> >>>> >>>>>> data > >> >>>> >>>>>> fed in as a numpy array. This dataset acts just like > >> >>>> >>>>>> an > >> >>>> >>>>>> on-disk simulation > >> >>>> >>>>>> dataset, but doesn't come with the baggage of needing > >> >>>> >>>>>> to write > >> >>>> >>>>>> a custom > >> >>>> >>>>>> frontend to read a specific data format off disk. > >> >>>> >>>>>> > >> >>>> >>>>>> covering_grid is a way to generate uniform resolution > >> >>>> >>>>>> data > >> >>>> >>>>>> from > >> >>>> >>>>>> an AMR dataset. It acts like a python dictionary where > >> >>>> >>>>>> the > >> >>>> >>>>>> keys are field > >> >>>> >>>>>> names and returns 3D numpy arrays of whatever uniform > >> >>>> >>>>>> resolution you specify > >> >>>> >>>>>> when you create the covering_grid. > >> >>>> >>>>>> > >> >>>> >>>>>> Note that if you're using load_particles all of your > >> >>>> >>>>>> data > >> >>>> >>>>>> needs > >> >>>> >>>>>> to live in memory. If your data is too big for that > >> >>>> >>>>>> you'll > >> >>>> >>>>>> need to write a > >> >>>> >>>>>> frontend for your data format or use a memmap to an > >> >>>> >>>>>> on-disk > >> >>>> >>>>>> file somehow. > >> >>>> >>>>>> I'm not an expert on that but others on the list > >> >>>> >>>>>> should be > >> >>>> >>>>>> able to help out. > >> >>>> >>>>>> > >> >>>> >>>>>> Hope that gets you well on your way :) > >> >>>> >>>>>> > >> >>>> >>>>>> -Nathan > >> >>>> >>>>>> > >> >>>> >>>>>> > >> >>>> >>>>>> On Thu, Jun 5, 2014 at 5:04 PM, Desika Narayanan > >> >>>> >>>>>> <dnarayan@haverford.edu> wrote: > >> >>>> >>>>>>> > >> >>>> >>>>>>> Hey Brendan, > >> >>>> >>>>>>> > >> >>>> >>>>>>> A couple of extra tools you might find helpful in > >> >>>> >>>>>>> conjunction > >> >>>> >>>>>>> with Nathan's example of depositing the particles > >> >>>> >>>>>>> onto an > >> >>>> >>>>>>> octree are at: > >> >>>> >>>>>>> > >> >>>> >>>>>>> http://paste.yt-project.org/show/4737/ > >> >>>> >>>>>>> > >> >>>> >>>>>>> Where I load a gadget snapshot, and then recover the > >> >>>> >>>>>>> coordinates > >> >>>> >>>>>>> and width of each cell. > >> >>>> >>>>>>> > >> >>>> >>>>>>> In response to your last question - the particles are > >> >>>> >>>>>>> deposited > >> >>>> >>>>>>> into an octree grid (so, you'll see that the cell > >> >>>> >>>>>>> sizes > >> >>>> >>>>>>> aren't all the same > >> >>>> >>>>>>> size). I don't know if depositing onto a regular > >> >>>> >>>>>>> NxNxN mesh > >> >>>> >>>>>>> is possible, > >> >>>> >>>>>>> though would be interested to hear if so. > >> >>>> >>>>>>> > >> >>>> >>>>>>> -d > >> >>>> >>>>>>> > >> >>>> >>>>>>> > >> >>>> >>>>>>> On Thu, Jun 5, 2014 at 7:58 PM, Brendan Griffen > >> >>>> >>>>>>> <brendan.f.griffen@gmail.com> wrote: > >> >>>> >>>>>>>> > >> >>>> >>>>>>>> Thanks. I'll get the "bleeding edge" version first > >> >>>> >>>>>>>> then try > >> >>>> >>>>>>>> your suggestions. Though I want to return the NxNxN > >> >>>> >>>>>>>> array > >> >>>> >>>>>>>> and be able to > >> >>>> >>>>>>>> write this mesh to a file. It is *only* using the > >> >>>> >>>>>>>> cic part > >> >>>> >>>>>>>> of yt and it > >> >>>> >>>>>>>> should return the mesh to be written? Just wanted to > >> >>>> >>>>>>>> clarify? > >> >>>> >>>>>>>> > >> >>>> >>>>>>>> Thanks. > >> >>>> >>>>>>>> Brendan > >> >>>> >>>>>>>> > >> >>>> >>>>>>>> > >> >>>> >>>>>>>> On Thu, Jun 5, 2014 at 6:49 PM, Nathan Goldbaum > >> >>>> >>>>>>>> <nathan12343@gmail.com> wrote: > >> >>>> >>>>>>>>> > >> >>>> >>>>>>>>> > >> >>>> >>>>>>>>> > >> >>>> >>>>>>>>> > >> >>>> >>>>>>>>> On Thu, Jun 5, 2014 at 3:36 PM, John ZuHone > >> >>>> >>>>>>>>> <jzuhone@gmail.com> wrote: > >> >>>> >>>>>>>>>> > >> >>>> >>>>>>>>>> Hi Brendan, > >> >>>> >>>>>>>>>> > >> >>>> >>>>>>>>>> Which version of yt are you using? > >> >>>> >>>>>>>>>> > >> >>>> >>>>>>>>>> If you're using 3.0, this is actually fairly easy. > >> >>>> >>>>>>>>>> If you > >> >>>> >>>>>>>>>> look in yt.fields.particle_fields.py, around line > >> >>>> >>>>>>>>>> 85, you > >> >>>> >>>>>>>>>> can see how this > >> >>>> >>>>>>>>>> is done for the "particle_density" and > >> >>>> >>>>>>>>>> "particle_mass" > >> >>>> >>>>>>>>>> fields. Basically you > >> >>>> >>>>>>>>>> can call a "deposit" method which takes the > >> >>>> >>>>>>>>>> particle field > >> >>>> >>>>>>>>>> quantity you want > >> >>>> >>>>>>>>>> deposited and deposits it into cells. The > >> >>>> >>>>>>>>>> underlying > >> >>>> >>>>>>>>>> calculation is done > >> >>>> >>>>>>>>>> using Cython, so it's fast. > >> >>>> >>>>>>>>> > >> >>>> >>>>>>>>> > >> >>>> >>>>>>>>> And you shouldn't ever actually need to call these > >> >>>> >>>>>>>>> "deposit" > >> >>>> >>>>>>>>> functions, since "deposit" is exposed as a field > >> >>>> >>>>>>>>> type for > >> >>>> >>>>>>>>> all datasets that > >> >>>> >>>>>>>>> contain particles. > >> >>>> >>>>>>>>> > >> >>>> >>>>>>>>> Here is a notebook that does this for Enzo AMR > >> >>>> >>>>>>>>> data: > >> >>>> >>>>>>>>> > >> >>>> >>>>>>>>> > >> >>>> >>>>>>>>> > >> >>>> >>>>>>>>> > >> >>>> >>>>>>>>> http://nbviewer.ipython.org/gist/ngoldbaum/5e19e4e6cc2bf330149c > >> >>>> >>>>>>>>> > >> >>>> >>>>>>>>> This dataset contains about a million particles and > >> >>>> >>>>>>>>> generates > >> >>>> >>>>>>>>> a CIC deposition for the whole domain in about 6 > >> >>>> >>>>>>>>> seconds > >> >>>> >>>>>>>>> from a cold start. > >> >>>> >>>>>>>>> > >> >>>> >>>>>>>>>> > >> >>>> >>>>>>>>>> > >> >>>> >>>>>>>>>> If you're using 2.x, then you can do the same > >> >>>> >>>>>>>>>> thing, but > >> >>>> >>>>>>>>>> it's > >> >>>> >>>>>>>>>> not as straightforward. You can see how this works > >> >>>> >>>>>>>>>> in > >> >>>> >>>>>>>>>> yt.data_objects.universal_fields.py, around line > >> >>>> >>>>>>>>>> 986, > >> >>>> >>>>>>>>>> where the > >> >>>> >>>>>>>>>> "particle_density" field is defined. Basically, it > >> >>>> >>>>>>>>>> calls > >> >>>> >>>>>>>>>> CICDeposit_3, which > >> >>>> >>>>>>>>>> is in yt.utilities.lib.CICDeposit.pyx. > >> >>>> >>>>>>>>>> > >> >>>> >>>>>>>>>> Let me know if you need any more clarification. > >> >>>> >>>>>>>>>> > >> >>>> >>>>>>>>>> Best, > >> >>>> >>>>>>>>>> > >> >>>> >>>>>>>>>> John Z > >> >>>> >>>>>>>>>> > >> >>>> >>>>>>>>>> On Jun 5, 2014, at 6:07 PM, Brendan Griffen > >> >>>> >>>>>>>>>> <brendan.f.griffen@gmail.com> wrote: > >> >>>> >>>>>>>>>> > >> >>>> >>>>>>>>>> > Hi, > >> >>>> >>>>>>>>>> > > >> >>>> >>>>>>>>>> > I was wondering if there were any Cython > >> >>>> >>>>>>>>>> > routines within > >> >>>> >>>>>>>>>> > yt > >> >>>> >>>>>>>>>> > which takes particle data and converts it into a > >> >>>> >>>>>>>>>> > cloud-in-cell based mesh > >> >>>> >>>>>>>>>> > which can be written to a file of my choosing. > >> >>>> >>>>>>>>> > >> >>>> >>>>>>>>> > >> >>>> >>>>>>>>> What sort of mesh were you looking for? yt will > >> >>>> >>>>>>>>> internally > >> >>>> >>>>>>>>> construct an octree if it is fed particle data. > >> >>>> >>>>>>>>> I'm not > >> >>>> >>>>>>>>> sure whether this > >> >>>> >>>>>>>>> octree can be saved to disk for later analysis. > >> >>>> >>>>>>>>> > >> >>>> >>>>>>>>> It's also possible to create a uniform resolution > >> >>>> >>>>>>>>> covering > >> >>>> >>>>>>>>> grid containing field data for a deposited > >> >>>> >>>>>>>>> quantity, which > >> >>>> >>>>>>>>> can be quite > >> >>>> >>>>>>>>> easily saved to disk in a number of ways. > >> >>>> >>>>>>>>> > >> >>>> >>>>>>>>>> > >> >>>> >>>>>>>>>> I heard a while ago there was some such > >> >>>> >>>>>>>>>> functionality but > >> >>>> >>>>>>>>>> it > >> >>>> >>>>>>>>>> could be too far down the yt rabbit hole to be > >> >>>> >>>>>>>>>> used as a > >> >>>> >>>>>>>>>> standalone? Is this > >> >>>> >>>>>>>>>> true? I have my own Python code for doing it but > >> >>>> >>>>>>>>>> it just > >> >>>> >>>>>>>>>> isn't fast enough > >> >>>> >>>>>>>>>> and thought I'd ask the yt community if there were > >> >>>> >>>>>>>>>> any > >> >>>> >>>>>>>>>> wrapper tools > >> >>>> >>>>>>>>>> available to boost the speed. > >> >>>> >>>>>>>>>> > > >> >>>> >>>>>>>>>> > Thanks. > >> >>>> >>>>>>>>>> > Brendan > >> >>>> >>>>>>>>>> > _______________________________________________ > >> >>>> >>>>>>>>>> > yt-users mailing list > >> >>>> >>>>>>>>>> > yt-users@lists.spacepope.org > >> >>>> >>>>>>>>>> > > >> >>>> >>>>>>>>>> > > >> >>>> >>>>>>>>>> > > >> >>>> >>>>>>>>>> > http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org > >> >>>> >>>>>>>>>> > >> >>>> >>>>>>>>>> _______________________________________________ > >> >>>> >>>>>>>>>> yt-users mailing list > >> >>>> >>>>>>>>>> yt-users@lists.spacepope.org > >> >>>> >>>>>>>>>> > >> >>>> >>>>>>>>>> > >> >>>> >>>>>>>>>> > >> >>>> >>>>>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org > >> >>>> >>>>>>>>> > >> >>>> >>>>>>>>> > >> >>>> >>>>>>>>> > >> >>>> >>>>>>>>> _______________________________________________ > >> >>>> >>>>>>>>> yt-users mailing list > >> >>>> >>>>>>>>> yt-users@lists.spacepope.org > >> >>>> >>>>>>>>> > >> >>>> >>>>>>>>> > >> >>>> >>>>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org > >> >>>> >>>>>>>>> > >> >>>> >>>>>>>> > >> >>>> >>>>>>>> > >> >>>> >>>>>>>> _______________________________________________ > >> >>>> >>>>>>>> yt-users mailing list > >> >>>> >>>>>>>> yt-users@lists.spacepope.org > >> >>>> >>>>>>>> > >> >>>> >>>>>>>> > >> >>>> >>>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org > >> >>>> >>>>>>>> > >> >>>> >>>>>>> > >> >>>> >>>>>>> > >> >>>> >>>>>>> _______________________________________________ > >> >>>> >>>>>>> yt-users mailing list > >> >>>> >>>>>>> yt-users@lists.spacepope.org > >> >>>> >>>>>>> > >> >>>> >>>>>>> > >> >>>> >>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org > >> >>>> >>>>>>> > >> >>>> >>>>>> > >> >>>> >>>>>> > >> >>>> >>>>>> _______________________________________________ > >> >>>> >>>>>> yt-users mailing list > >> >>>> >>>>>> yt-users@lists.spacepope.org > >> >>>> >>>>>> > >> >>>> >>>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org > >> >>>> >>>>>> > >> >>>> >>>>> > >> >>>> >>>>> > >> >>>> >>>>> _______________________________________________ > >> >>>> >>>>> yt-users mailing list > >> >>>> >>>>> yt-users@lists.spacepope.org > >> >>>> >>>>> > >> >>>> >>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org > >> >>>> >>>>> > >> >>>> >>>> > >> >>>> >>>> > >> >>>> >>>> _______________________________________________ > >> >>>> >>>> yt-users mailing list > >> >>>> >>>> yt-users@lists.spacepope.org > >> >>>> >>>> > >> >>>> >>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org > >> >>>> >>>> > >> >>>> >>> > >> >>>> >>> > >> >>>> >>> _______________________________________________ > >> >>>> >>> yt-users mailing list > >> >>>> >>> yt-users@lists.spacepope.org > >> >>>> >>> > >> >>>> >>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org > >> >>>> >>> > >> >>>> >> > >> >>>> >> > >> >>>> >> _______________________________________________ > >> >>>> >> yt-users mailing list > >> >>>> >> yt-users@lists.spacepope.org > >> >>>> >> > >> >>>> >> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org > >> >>>> >> > >> >>>> > > >> >>>> > > >> >>>> > _______________________________________________ > >> >>>> > yt-users mailing list > >> >>>> > yt-users@lists.spacepope.org > >> >>>> > > >> >>>> > http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org > >> >>>> > > >> >>>> > >> >>>> > >> >>>> _______________________________________________ > >> >>>> yt-users mailing list > >> >>>> yt-users@lists.spacepope.org > >> >>>> > >> >>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org > >> >>>> > >> >>> > >> >> > >> >> > >> >> _______________________________________________ > >> >> yt-users mailing list > >> >> yt-users@lists.spacepope.org > >> >> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org > >> >> > >> > > >> > > >> > _______________________________________________ > >> > yt-users mailing list > >> > yt-users@lists.spacepope.org > >> > http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org > >> > > >> _______________________________________________ > >> yt-users mailing list > >> yt-users@lists.spacepope.org > >> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org > > > > > > > > _______________________________________________ > > yt-users mailing list > > yt-users@lists.spacepope.org > > http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org > > > _______________________________________________ > yt-users mailing list > yt-users@lists.spacepope.org > http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
Thanks very much Matt! I've improved things at my end as well with my script (non-yt related). Hopefully the confluence of the two changes will enabled a successful run. Thanks. Brendan On Tue, Jun 10, 2014 at 3:57 PM, Matthew Turk <matthewturk@gmail.com> wrote:
Hi Brendan,
I think I've found the problem -- volume is making too many copies. I'm working on a fix.
Yes, Matt and I are chatting off-list about this. I'll have more to say later hopefully.
On Mon, Jun 9, 2014 at 10:48 AM, Brendan Griffen <brendan.f.griffen@gmail.com> wrote:
Hi Nathan, I'm not sure if you missed it but earlier Matt asked if there was some copying which shouldn't be happening?
Brendan
On Mon, Jun 9, 2014 at 1:22 PM, Nathan Goldbaum <nathan12343@gmail.com> wrote:
On Mon, Jun 9, 2014 at 12:32 AM, Brendan Griffen <brendan.f.griffen@gmail.com> wrote:
Thanks Nathan. Comparing the two though:
26 1536.664 MiB 1130.480 MiB uniform_array = grid_object['deposit', 'all_cic']
24 1285.703 MiB 890.539 MiB cic_density = ad["deposit", "all_cic"]
Isn't it the uniform grid which uses the most memory? I'm running the profiling now and will get back to you. I hope in spite of the fact
On Mon, Jun 9, 2014 at 12:56 PM, Nathan Goldbaum <nathan12343@gmail.com> wrote: that it
does crash it will still give me some useful output.
That's true - I was thinking it might be more memory efficient since there is no need to construct the covering grid in addition to the octree.
Brendan
On Mon, Jun 9, 2014 at 2:05 AM, Nathan Goldbaum <
nathan12343@gmail.com>
wrote:
Hey Brendan,
Could you try running your script using the memory_profiler module on pypi?
Here's an example script that uses the memory profiler: http://paste.yt-project.org/show/4748/
and the output for that script:
http://paste.yt-project.org/show/4749/
For what it's worth, it does indeed look like matt's suggestion to
use
load_uniform_grid is an option, and might be more memory efficient in the end since you will go directly to the uniform grid you want without creating an octree. Here's an example: http://paste.yt-project.org/show/4750/
Here's the memory usage information for that example: http://paste.yt-project.org/show/4751/
I used a 256^3 uniform grid with normally distributed random data - I'm not sure whether it will also be more memory efficient in your case.
On Sun, Jun 8, 2014 at 9:32 PM, Brendan Griffen <brendan.f.griffen@gmail.com> wrote: > > This is the full error if it helps at all? It is indeed, loading in > all of the quantities. > > Loading particles... > --> Loading particle type: 1 > yt : [INFO ] 2014-06-08 15:12:05,540 Parameters: current_time > = 0.0 > yt : [INFO ] 2014-06-08 15:12:05,540 Parameters: domain_dimensions > = [2 2 2] > yt : [INFO ] 2014-06-08 15:12:05,541 Parameters: domain_left_edge > = [ 0. 0. 0.] > yt : [INFO ] 2014-06-08 15:12:05,542 Parameters: domain_right_edge > = [ 100. 100. 100.] > yt : [INFO ] 2014-06-08 15:12:05,542 Parameters: > cosmological_simulation = 0.0 > yt : [INFO ] 2014-06-08 15:12:05,548 Allocating for 1.074e+09 > particles > yt : [INFO ] 2014-06-08 15:16:16,195 Identified 7.584e+07 octs > yt : [INFO ] 2014-06-08 15:16:16,299 Loading field plugins. > yt : [INFO ] 2014-06-08 15:16:16,299 Loaded angular_momentum (8 > new fields) > yt : [INFO ] 2014-06-08 15:16:16,299 Loaded astro (14 new fields) > yt : [INFO ] 2014-06-08 15:16:16,300 Loaded cosmology (20 new > fields) > yt : [INFO ] 2014-06-08 15:16:16,300 Loaded fluid (56 new fields) > yt : [INFO ] 2014-06-08 15:16:16,301 Loaded fluid_vector (88 new > fields) > yt : [INFO ] 2014-06-08 15:16:16,301 Loaded geometric (103 new > fields) > yt : [INFO ] 2014-06-08 15:16:16,301 Loaded local (103 new fields) > yt : [INFO ] 2014-06-08 15:16:16,302 Loaded magnetic_field (109 > new fields) > yt : [INFO ] 2014-06-08 15:16:16,302 Loaded species (109 new > fields) > >
> MemoryError Traceback (most recent call > last) > > /nfs/blank/h4231/bgriffen/data/lib/yt-x86_64/lib/python2.7/site-packages/IPython/utils/py3compat.pyc > in execfile(fname, *where) > 202 else: > 203 filename = fname > --> 204 __builtin__.execfile(filename, *where) > > /nfs/blank/h4231/bgriffen/work/projects/caterpillar/c2ray/cic/ytcic.py > in <module>() > 99 slc.set_figure_size(4) > 100 slc.save() > --> 101 > 102 for ndim in dimlist: > 103 print > > > /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc > in __getitem__(self, key) > 218 return self.field_data[f] > 219 else: > --> 220 self.get_data(f) > 221 # fi.units is the unit expression string. We depend on > the registry > 222 # hanging off the dataset to define this unit object. > > > /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc > in get_data(self, fields) > 627 > 628 fields_to_generate += gen_fluids + gen_particles > --> 629 self._generate_fields(fields_to_generate) > 630 > 631 def _generate_fields(self, fields_to_generate): > > > /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc > in _generate_fields(self, fields_to_generate) > 644 fi = self.pf._get_field_info(*field) > 645 try: > --> 646 fd = self._generate_field(field) > 647 if type(fd) == np.ndarray: > 648 fd = self.pf.arr(fd, fi.units) > > > /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc > in _generate_field(self, field) > 255 tr = self._generate_particle_field(field) > 256 else: > --> 257 tr = self._generate_fluid_field(field) > 258 if tr is None: > 259 raise YTCouldNotGenerateField(field, self.pf) > > > /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc > in _generate_fluid_field(self, field) > 273 finfo.check_available(gen_obj) > 274 except NeedsGridType as ngt_exception: > --> 275 rv = self._generate_spatial_fluid(field, > ngt_exception.ghost_zones) > 276 else: > 277 rv = finfo(gen_obj) > > > /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc > in _generate_spatial_fluid(self, field, ngz) > 289 o = self._current_chunk.objs[0] > 290 with o._activate_cache(): > --> 291 ind += o.select(self.selector, > self[field], rv, ind) > 292 else: > 293 chunks = self.index._chunk(self, "spatial", ngz = > ngz) > > > /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc > in __getitem__(self, key) > 218 return self.field_data[f] > 219 else: > --> 220 self.get_data(f) > 221 # fi.units is the unit expression string. We depend on > the registry > 222 # hanging off the dataset to define this unit object. > > > /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc > in get_data(self, fields) > 627 > 628 fields_to_generate += gen_fluids + gen_particles > --> 629 self._generate_fields(fields_to_generate) > 630 > 631 def _generate_fields(self, fields_to_generate): > > > /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc > in _generate_fields(self, fields_to_generate) > 644 fi = self.pf._get_field_info(*field) > 645 try: > --> 646 fd = self._generate_field(field) > 647 if type(fd) == np.ndarray: > 648 fd = self.pf.arr(fd, fi.units) > > > /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc > in _generate_field(self, field) > 255 tr = self._generate_particle_field(field) > 256 else: > --> 257 tr = self._generate_fluid_field(field) > 258 if tr is None: > 259 raise YTCouldNotGenerateField(field, self.pf) > > > /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc > in _generate_fluid_field(self, field) > 275 rv = self._generate_spatial_fluid(field, > ngt_exception.ghost_zones) > 276 else: > --> 277 rv = finfo(gen_obj) > 278 return rv > 279 > > > /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/fields/derived_field.pyc > in __call__(self, data) > 176 "for %s" % (self.name,)) > 177 with self.unit_registry(data): > --> 178 dd = self._function(self, data) > 179 for field_name in data.keys(): > 180 if field_name not in original_fields: > > > /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/fields/particle_fields.pyc > in particle_cic(field, data) > 113 def particle_cic(field, data): > 114 pos = data[ptype, coord_name] > --> 115 d = data.deposit(pos, [data[ptype, mass_name]], method > = "cic") > 116 d = data.apply_units(d, data[ptype, mass_name].units) > 117 d /= data["index", "cell_volume"] > > > /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/octree_subset.pyc > in deposit(self, positions, fields, method) > 167 mylog.debug("Depositing %s (%s^3) particles into %s > Octs", > 168 positions.shape[0], positions.shape[0]**0.3333333, > nvals[-1]) > --> 169 pos = > np.asarray(positions.convert_to_units("code_length"), > 170 dtype="float64") > 171 # We should not need the following if we know in > advance all our fields > > /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/units/yt_array.pyc > in convert_to_units(self, units) > 366 > 367 self.units = new_units > --> 368 self *= conversion_factor > 369 return self > 370 > > /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/units/yt_array.pyc > in __imul__(self, other) > 667 """ See __mul__. """ > 668 oth = sanitize_units_mul(self, other) > --> 669 return np.multiply(self, oth, out=self) > 670 > 671 def __div__(self, right_object): > > /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/units/yt_array.pyc > in __array_wrap__(self, out_arr, context) > 966 # casting to YTArray avoids creating a > YTQuantity with size > 1 > 967 return YTArray(np.array(out_arr, unit)) > --> 968 return ret_class(np.array(out_arr), unit) > 969 > 970 > > MemoryError: > > > On Sun, Jun 8, 2014 at 10:50 PM, Matthew Turk < matthewturk@gmail.com> > wrote: >> >> Hi Brendan, >> >> On Sun, Jun 8, 2014 at 9:21 PM, Brendan Griffen >> <brendan.f.griffen@gmail.com> wrote: >> > Hi Matt, >> > >> > Thanks for your detailed email. Forgive my naivety but why do you >> > need the >> > oct-tree in the first place? I have a my own fortran code for >> > constructing a >> > cloud in cell mesh and it uses very little overhead (just the n^3 >> > grid and >> > the particle data itself). I then calculate the dx,dy,dzs to the >> > nearest 8 >> > grid points and distribute accordingly in a omp loop which is done >> > in a >> > fraction of a second. Does the situation with yt come about (oct >> > tree etc.) >> > necessarily because of the way it handles particle data? Is it >> > essentially >> > used to map the particles to domains in the grid or something? >> >> That's not naive at all. There are two reasons -- >> >> 1) The octree is used for indexing for neighbor lookups and >> early-termination of region selection for particles >> 2) The octree is used to estimate the "required resolution" for any >> operation that requires a space-filling value. (i.e., any time that >> a >> particle becomes a volume.) >> >> Projections in yt are adaptive, in that they project down to the >> finest appropriate resolution. There's also the "arbitrary_grid" >> operation, which does precisely what you're describing, but as it >> stands right now the octree gets constructed at time of instantiation >> of the indexing system. Thinking it over, you may be able to avoid >> that completely by not using load_particles and instead using >> load_uniform_grid and supplying your desired dimensions. The field >> names should be the same. >> >> > >> > The machine I max memory on has 128GB and the snapshots are using >> > 1024^3 >> > particles. Do you have any idea of how much memory the oct-tree >> > uses as a >> > function of particle/grid number? I am going to try on a 256GB >> > machine >> > (though this is a bit of a hassle). I'll see how I go. >> >> I am disappointed that it's blowing out your RAM. This week I will >> try to get some memory profiling done. Could you file a bug to this >> effect, which will help me track it? Peak memory usage during >> indexing should only be 64 bits * Nparticles, unless you're using >> load_particles, in which case all the fields will *also* have to be >> in >> memory. It's about 8 gigabytes per field. So, I think there's >> something going wrong. >> >> > >> > Thanks. >> > >> > Brendan >> > >> > >> > On Sun, Jun 8, 2014 at 6:25 PM, Matthew Turk >> > <matthewturk@gmail.com> wrote: >> >> >> >> Hi all, >> >> >> >> I feel like I owe a brief explanation of why things are tricky >> >> right >> >> now, what we're planning on doing, and how we're experimenting and >> >> developing. >> >> >> >> Presently, the particle geometry handlers build a single mesh from >> >> all >> >> particles in the dataset, along with a coarse bitmask that >> >> correlates >> >> files to regions in the domain. This requires the allocation of a >> >> single int64 array of size Nparticles, which is sorted in place >> >> and >> >> then fed into an octree construction algorithm that then spits >> >> back >> >> out the mesh. Each octree component contains 3 64-bit integers >> >> and >> >> eitehr a void pointer or a pointer to eight other octs. >> >> Increasing >> >> n_ref decreases the number of octs in this mesh; when smoothing >> >> operaitons are conducted, a second "index" mesh is created for >> >> looking >> >> up particles near mesh points. Mesh points are used for adaptive >> >> resolution smoothing and other "deposit particles on the grid >> >> somehow" >> >> operations (including SPH kernel). >> >> >> >> Anyway, because right now it requires a global mesh to be >> >> constructed, >> >> this is expensive and requires holding a 64-bit integer in memory >> >> for >> >> each particle. I think if you're loading the particles in >> >> differently >> >> there is some additional overhead as well, but I'm still a bit >> >> surprised you OOM on a 1024^3 dataset. >> >> >> >> In general, we don't *need* this global mesh; is can be >> >> constructed as >> >> required, which would speed up both the initial index phase as >> >> well as >> >> the final meshing process. I got about 50% of the way to >> >> implementing >> >> this last fall, but because of various concerns and deadlines I >> >> haven't finished it. I intend to get back to it probably in July, >> >> right after we put out a 3.0, so that we can have it in time for >> >> 3.1. >> >> In principle this will make the particle codes much more similar >> >> to >> >> ARTIO, in that the mesh will be constructed only as required and >> >> discarded when no longer required, which will make them much more >> >> memory efficient. >> >> >> >> But, getting a single mesh for extremely large data is a very high >> >> priority; right now for the 10240^3 run we've been loading up >> >> individual sub-chunks, which I want to stop doing. >> >> >> >> From the technical perspective, these are the things that need to >> >> happen on the yt side for particle datasets to move to this "lazy" >> >> mode of loading; most of this is based on things learned from 2HOT >> >> and >> >> ARTIO, and will involve converting to a forest-of-octrees. >> >> >> >> * Split into spatially-organized subchunks of >> >> ParticleOctreeSubset >> >> objects, such that these map 1:Nfiles, and that can be constructed >> >> on >> >> the fly. >> >> * Construct a dual-mesh of the bitmask "ParticleRegion" object >> >> that >> >> will help with identifying neighbors to a given oct cell, so that >> >> if >> >> we're inside one octree we know which neighbor octrees to grab if >> >> we >> >> need particles for smoothing things (fast boundary particle >> >> identification is later down the road) >> >> * Parallel sort of particles, or using the parallel ring >> >> function; >> >> may not be necessary after all >> >> >> >> All of this is doable, and I'd be happy to work with people if >> >> they'd >> >> like to take a shot at implementing it, but I've mostly put it on >> >> my >> >> list for post-3.0. >> >> >> >> -Matt >> >> >> >> On Sun, Jun 8, 2014 at 2:43 PM, Nathan Goldbaum >> >> <nathan12343@gmail.com> >> >> wrote: >> >> > >> >> > >> >> > >> >> > On Sun, Jun 8, 2014 at 12:27 PM, Brendan Griffen >> >> > <brendan.f.griffen@gmail.com> wrote: >> >> >> >> >> >> Also, how do I construct just a zero filled yt array with >> >> >> dimensions >> >> >> (ndim,ndim,ndim)? Thanks >> >> > >> >> > >> >> > >> >> > from yt import YTArray >> >> > from numpy import np >> >> > >> >> > arr = YTArray(np.zeros([ndim, ndim, ndim]), >> >> > input_units=units_string) >> >> > >> >> > or alternatively: >> >> > >> >> > from yt.units import kiloparsec >> >> > >> >> > arr = kiloparsec*np.zeros([ndim, ndim, ndim]) >> >> > >> >> > it doesn't have to be kiloparsec - you can compose the units you >> >> > want >> >> > out of >> >> > any of the unit symbols that live in yt.units. >> >> > >> >> > See this page for a ton more detail about yt's new unit system: >> >> > http://yt-project.org/docs/dev-3.0/analyzing/units/index.html >> >> > >> >> >> >> >> >> >> >> >> Brendan >> >> >> >> >> >> >> >> >> On Sun, Jun 8, 2014 at 3:26 PM, Brendan Griffen >> >> >> <brendan.f.griffen@gmail.com> wrote: >> >> >>> >> >> >>> Hi, >> >> >>> >> >> >>> Since I get memory errors. Could I not just read in the blocks >> >> >>> of the >> >> >>> output individually then basically stack the mesh each time. >> >> >>> That way >> >> >>> not >> >> >>> every single particle of the snapshot has to be loaded at the >> >> >>> same >> >> >>> time. >> >> >>> Would that just be a case of doing >> >> >>> >> >> >>> level = int(math.log(ndim,2)) >> >> >>> cg = ds.covering_grid(level=level, >> >> >>> left_edge=[0,0,0],dims=[ndim,ndim,ndim]) >> >> >>> arr = cg['deposit', 'all_density'] >> >> >>> arrall += arr >> >> >>> >> >> >>> in a loop over each HDF5 block? >> >> > >> >> > >> >> > It's likely that the memory use is dominated by the octree >> >> > rather than >> >> > the >> >> > covering grid. This is with 1024^3 particles, correct? >> >> > >> >> > You can probably significantly reduce the memory used by the >> >> > octree by >> >> > increasing n_ref in the call to load_particles. >> >> > >> >> > See this page for more detail about load_particles: >> >> > >> >> > >> >> > http://yt-project.org/docs/dev-3.0/examining/loading_data.html#generic-parti... >> >> > >> >> > Larger n_ref means fewer octree cells (lower resolution), but it >> >> > also >> >> > means >> >> > lower poisson noise and lower memory use. >> >> > >> >> > Alternatively, as Matt suggested, you could break your 1024^3 >> >> > ensemble >> >> > of >> >> > particles up into chunks, loop over the chunk, creating a >> >> > particle >> >> > octree >> >> > and then a covering grid for each subset of the particles. Your >> >> > final >> >> > covering grid is just the sub of the covering grids for each >> >> > subset of >> >> > particles. >> >> > >> >> >>> >> >> >>> >> >> >>> Thanks. >> >> >>> Brendan >> >> >>> >> >> >>> >> >> >>> >> >> >>> >> >> >>> On Fri, Jun 6, 2014 at 7:26 PM, Matthew Turk >> >> >>> <matthewturk@gmail.com> >> >> >>> wrote: >> >> >>>> >> >> >>>> >> >> >>>> On Jun 6, 2014 4:54 PM, "Brendan Griffen" >> >> >>>> <brendan.f.griffen@gmail.com> >> >> >>>> wrote: >> >> >>>> > >> >> >>>> > OK great. It is very low resolution but it worked. Thanks >> >> >>>> > for all >> >> >>>> > your >> >> >>>> > help. My higher resolution run 1024^3 in 100 Mpc seems to >> >> >>>> > crash on >> >> >>>> > 128GB >> >> >>>> > memory machine. I might have to look elsewhere. >> >> >>>> > >> >> >>>> >> >> >>>> If you are looking for low resolution extraction you can >> >> >>>> tune the >> >> >>>> memory usage by changing the parameter n_ref to something >> >> >>>> higher. >> >> >>>> >> >> >>>> Supporting extremely large datasets in a single mesh is on >> >> >>>> the >> >> >>>> roadmap >> >> >>>> for the late summer or fall, after a 3.0 release goes out. >> >> >>>> For now >> >> >>>> you can >> >> >>>> also extract before you load in; this is sort of how we are >> >> >>>> supporting an >> >> >>>> INCITE project with very large particle counts. >> >> >>>> >> >> >>>> >> >> >>>> > Also, I normally use Canopy distribution but I just use an >> >> >>>> > alias to >> >> >>>> > loadyt which erases my PYTHONPATH and I can't access scipy >> >> >>>> > and a >> >> >>>> > few other >> >> >>>> > libraries any more. What is the best practice here? Should >> >> >>>> > I just >> >> >>>> > manually >> >> >>>> > export PYTHONPATH and point to the libraries need in canopy >> >> >>>> > or can >> >> >>>> > they play >> >> >>>> > nice together? >> >> >>>> > >> >> >>>> > Thanks. >> >> >>>> > >> >> >>>> > BG >> >> >>>> > >> >> >>>> > >> >> >>>> > On Fri, Jun 6, 2014 at 2:54 PM, Nathan Goldbaum >> >> >>>> > <nathan12343@gmail.com> wrote: >> >> >>>> >> >> >> >>>> >> >> >> >>>> >> >> >> >>>> >> >> >> >>>> >> On Fri, Jun 6, 2014 at 11:48 AM, Brendan Griffen >> >> >>>> >> <brendan.f.griffen@gmail.com> wrote: >> >> >>>> >>> >> >> >>>> >>> OK great. Thanks. I just wanted a homogeneous mesh. 512^3 >> >> >>>> >>> with no >> >> >>>> >>> nesting of any kind. Though when I plot the image it >> >> >>>> >>> looks like >> >> >>>> >>> it is >> >> >>>> >>> assigning particles incorrectly (low resolution on the >> >> >>>> >>> outside). >> >> >>>> >>> This is >> >> >>>> >>> just a test image. >> >> >>>> >>> >> >> >>>> >> >> >> >>>> >> The SlicePlot is visualizing the octree so there is less >> >> >>>> >> resolution >> >> >>>> >> where there are fewer particles. If you want to visualize >> >> >>>> >> the >> >> >>>> >> covering grid >> >> >>>> >> you're going to need to visualize that separately. >> >> >>>> >> >> >> >>>> >>> >> >> >>>> >>> ds = yt.load_particles(data, length_unit=3.08e24, >> >> >>>> >>> mass_unit=1.9891e33,bbox=bbox) >> >> >>>> >>> >> >> >>>> >>> ad = ds.all_data() >> >> >>>> >>> print ad['deposit', 'all_cic'] >> >> >>>> >>> slc = yt.SlicePlot(ds, 2, ('deposit', 'all_cic')) >> >> >>>> >>> slc.set_figure_size(4) >> >> >>>> >>> cg = ds.covering_grid(level=9, >> >> >>>> >>> left_edge=[0,0,0],dims=[512,512,512]) >> >> >>>> >>> >> >> >>>> >> >> >> >>>> >> To actually produce the uniform resolution ndarray, you're >> >> >>>> >> going >> >> >>>> >> to >> >> >>>> >> need to do something like: >> >> >>>> >> >> >> >>>> >> array = cg[('deposit', 'all_cic')] >> >> >>>> >> >> >> >>>> >> array will then be a 3D array you can do whatever you want >> >> >>>> >> with. >> >> >>>> >> By >> >> >>>> >> default it has units, but to strip them off you'll just >> >> >>>> >> need to >> >> >>>> >> cast to >> >> >>>> >> ndarray: >> >> >>>> >> >> >> >>>> >> array_without_units = array.v >> >> >>>> >> >> >> >>>> >> >> >> >>>> >>> >> >> >>>> >>> Also, is there a way to load multiple particle types? >> >> >>>> >>> >> >> >>>> >>> Do I just need to stack the particles into the array >> >> >>>> >>> here? >> >> >>>> >>> >> >> >>>> >>> data = {'particle_position_x': pos[:,0], >> >> >>>> >>> 'particle_position_y': pos[:,1], >> >> >>>> >>> 'particle_position_z': pos[:,2], >> >> >>>> >>> 'particle_mass': np.array([mpart]*npart)} >> >> >>>> >>> >> >> >>>> >>> Then feed it in as usual? >> >> >>>> >> >> >> >>>> >> >> >> >>>> >> That's right, although if the particle masses are >> >> >>>> >> different for >> >> >>>> >> the >> >> >>>> >> different particle types that code snippet will need to be >> >> >>>> >> generalized to >> >> >>>> >> handle that. >> >> >>>> >> >> >> >>>> >> I think in principle it should be possible to make >> >> >>>> >> load_particles >> >> >>>> >> handle different particle types just like an SPH dataset >> >> >>>> >> that >> >> >>>> >> contains >> >> >>>> >> multiple particle types, but right now that hasn't been >> >> >>>> >> implemented yet. >> >> >>>> >> >> >> >>>> >>> >> >> >>>> >>> >> >> >>>> >>> Brendan >> >> >>>> >>> >> >> >>>> >>> >> >> >>>> >>> On Thu, Jun 5, 2014 at 9:44 PM, Nathan Goldbaum >> >> >>>> >>> <nathan12343@gmail.com> wrote: >> >> >>>> >>>> >> >> >>>> >>>> That's right, you can set that via the bbox keyword >> >> >>>> >>>> parameter >> >> >>>> >>>> for >> >> >>>> >>>> load_particles. I'd urge you to take a look at the >> >> >>>> >>>> docstrings >> >> >>>> >>>> and source >> >> >>>> >>>> code for load_particles. >> >> >>>> >>>> >> >> >>>> >>>> >> >> >>>> >>>> On Thu, Jun 5, 2014 at 6:34 PM, Brendan Griffen >> >> >>>> >>>> <brendan.f.griffen@gmail.com> wrote: >> >> >>>> >>>>> >> >> >>>> >>>>> Thanks very much Nathan. I tried to load in my own data >> >> >>>> >>>>> but I >> >> >>>> >>>>> think there are too many particles or I have to >> >> >>>> >>>>> specifically >> >> >>>> >>>>> set the domain >> >> >>>> >>>>> size. >> >> >>>> >>>>> >> >> >>>> >>>>> In this area: >> >> >>>> >>>>> >> >> >>>> >>>>> data = {'particle_position_x': pos[:,0], >> >> >>>> >>>>> 'particle_position_y': pos[:,1], >> >> >>>> >>>>> 'particle_position_z': pos[:,2], >> >> >>>> >>>>> 'particle_mass': np.array([mpart]*npart)} >> >> >>>> >>>>> >> >> >>>> >>>>> ds = yt.load_particles(data, length_unit=3.08e24, >> >> >>>> >>>>> mass_unit=1.9891e36) >> >> >>>> >>>>> ad = ds.all_data() >> >> >>>> >>>>> print ad['deposit', 'all_cic'] >> >> >>>> >>>>> >> >> >>>> >>>>> In [3]: run ytcic.py >> >> >>>> >>>>> yt : [INFO ] 2014-06-05 21:29:06,183 Parameters: >> >> >>>> >>>>> current_time >> >> >>>> >>>>> = 0.0 >> >> >>>> >>>>> yt : [INFO ] 2014-06-05 21:29:06,183 Parameters: >> >> >>>> >>>>> domain_dimensions = [2 2 2] >> >> >>>> >>>>> yt : [INFO ] 2014-06-05 21:29:06,184 Parameters: >> >> >>>> >>>>> domain_left_edge = [ 0. 0. 0.] >> >> >>>> >>>>> yt : [INFO ] 2014-06-05 21:29:06,185 Parameters: >> >> >>>> >>>>> domain_right_edge = [ 1. 1. 1.] >> >> >>>> >>>>> yt : [INFO ] 2014-06-05 21:29:06,185 Parameters: >> >> >>>> >>>>> cosmological_simulation = 0.0 >> >> >>>> >>>>> yt : [INFO ] 2014-06-05 21:29:06,188 Allocating for >> >> >>>> >>>>> 1.342e+08 >> >> >>>> >>>>> particles >> >> >>>> >>>>> >> >> >>>> >>>>> >> >> >>>> >>>>> >> >> >>>> >>>>>
>> >> >>>> >>>>> 0.] >> >> >>>> >>>>> code_length and >> >> >>>> >>>>> [ 1. 1. 1.] code_length >> >> >>>> >>>>> >> >> >>>> >>>>> >> >> >>>> >>>>> >> >> >>>> >>>>> On Thu, Jun 5, 2014 at 8:22 PM, Nathan Goldbaum >> >> >>>> >>>>> <nathan12343@gmail.com> wrote: >> >> >>>> >>>>>> >> >> >>>> >>>>>> Here's a worked out example that does what you're >> >> >>>> >>>>>> looking for >> >> >>>> >>>>>> using a fake 1 million particle dataset: >> >> >>>> >>>>>> >> >> >>>> >>>>>> >> >> >>>> >>>>>> >> >> >>>> >>>>>> http://nbviewer.ipython.org/gist/ngoldbaum/546d37869aafe71cfe38 >> >> >>>> >>>>>> >> >> >>>> >>>>>> In this notebook I make use of two key yt features: >> >> >>>> >>>>>> `load_particles`, and `covering_grid`. >> >> >>>> >>>>>> >> >> >>>> >>>>>> load_particles creates a "stream" dataset based on >> >> >>>> >>>>>> in-memory >> >> >>>> >>>>>> data >> >> >>>> >>>>>> fed in as a numpy array. This dataset acts just like >> >> >>>> >>>>>> an >> >> >>>> >>>>>> on-disk simulation >> >> >>>> >>>>>> dataset, but doesn't come with the baggage of needing >> >> >>>> >>>>>> to write >> >> >>>> >>>>>> a custom >> >> >>>> >>>>>> frontend to read a specific data format off disk. >> >> >>>> >>>>>> >> >> >>>> >>>>>> covering_grid is a way to generate uniform resolution >> >> >>>> >>>>>> data >> >> >>>> >>>>>> from >> >> >>>> >>>>>> an AMR dataset. It acts like a python dictionary where >> >> >>>> >>>>>> the >> >> >>>> >>>>>> keys are field >> >> >>>> >>>>>> names and returns 3D numpy arrays of whatever uniform >> >> >>>> >>>>>> resolution you specify >> >> >>>> >>>>>> when you create the covering_grid. >> >> >>>> >>>>>> >> >> >>>> >>>>>> Note that if you're using load_particles all of your >> >> >>>> >>>>>> data >> >> >>>> >>>>>> needs >> >> >>>> >>>>>> to live in memory. If your data is too big for that >> >> >>>> >>>>>> you'll >> >> >>>> >>>>>> need to write a >> >> >>>> >>>>>> frontend for your data format or use a memmap to an >> >> >>>> >>>>>> on-disk >> >> >>>> >>>>>> file somehow. >> >> >>>> >>>>>> I'm not an expert on that but others on the list >> >> >>>> >>>>>> should be >> >> >>>> >>>>>> able to help out. >> >> >>>> >>>>>> >> >> >>>> >>>>>> Hope that gets you well on your way :) >> >> >>>> >>>>>> >> >> >>>> >>>>>> -Nathan >> >> >>>> >>>>>> >> >> >>>> >>>>>> >> >> >>>> >>>>>> On Thu, Jun 5, 2014 at 5:04 PM, Desika Narayanan >> >> >>>> >>>>>> <dnarayan@haverford.edu> wrote: >> >> >>>> >>>>>>> >> >> >>>> >>>>>>> Hey Brendan, >> >> >>>> >>>>>>> >> >> >>>> >>>>>>> A couple of extra tools you might find helpful in >> >> >>>> >>>>>>> conjunction >> >> >>>> >>>>>>> with Nathan's example of depositing the particles >> >> >>>> >>>>>>> onto an >> >> >>>> >>>>>>> octree are at: >> >> >>>> >>>>>>> >> >> >>>> >>>>>>> http://paste.yt-project.org/show/4737/ >> >> >>>> >>>>>>> >> >> >>>> >>>>>>> Where I load a gadget snapshot, and then recover the >> >> >>>> >>>>>>> coordinates >> >> >>>> >>>>>>> and width of each cell. >> >> >>>> >>>>>>> >> >> >>>> >>>>>>> In response to your last question - the particles are >> >> >>>> >>>>>>> deposited >> >> >>>> >>>>>>> into an octree grid (so, you'll see that the cell >> >> >>>> >>>>>>> sizes >> >> >>>> >>>>>>> aren't all the same >> >> >>>> >>>>>>> size). I don't know if depositing onto a regular >> >> >>>> >>>>>>> NxNxN mesh >> >> >>>> >>>>>>> is possible, >> >> >>>> >>>>>>> though would be interested to hear if so. >> >> >>>> >>>>>>> >> >> >>>> >>>>>>> -d >> >> >>>> >>>>>>> >> >> >>>> >>>>>>> >> >> >>>> >>>>>>> On Thu, Jun 5, 2014 at 7:58 PM, Brendan Griffen >> >> >>>> >>>>>>> <brendan.f.griffen@gmail.com> wrote: >> >> >>>> >>>>>>>> >> >> >>>> >>>>>>>> Thanks. I'll get the "bleeding edge" version first >> >> >>>> >>>>>>>> then try >> >> >>>> >>>>>>>> your suggestions. Though I want to return the NxNxN >> >> >>>> >>>>>>>> array >> >> >>>> >>>>>>>> and be able to >> >> >>>> >>>>>>>> write this mesh to a file. It is *only* using the >> >> >>>> >>>>>>>> cic part >> >> >>>> >>>>>>>> of yt and it >> >> >>>> >>>>>>>> should return the mesh to be written? Just wanted to >> >> >>>> >>>>>>>> clarify? >> >> >>>> >>>>>>>> >> >> >>>> >>>>>>>> Thanks. >> >> >>>> >>>>>>>> Brendan >> >> >>>> >>>>>>>> >> >> >>>> >>>>>>>> >> >> >>>> >>>>>>>> On Thu, Jun 5, 2014 at 6:49 PM, Nathan Goldbaum >> >> >>>> >>>>>>>> <nathan12343@gmail.com> wrote: >> >> >>>> >>>>>>>>> >> >> >>>> >>>>>>>>> >> >> >>>> >>>>>>>>> >> >> >>>> >>>>>>>>> >> >> >>>> >>>>>>>>> On Thu, Jun 5, 2014 at 3:36 PM, John ZuHone >> >> >>>> >>>>>>>>> <jzuhone@gmail.com> wrote: >> >> >>>> >>>>>>>>>> >> >> >>>> >>>>>>>>>> Hi Brendan, >> >> >>>> >>>>>>>>>> >> >> >>>> >>>>>>>>>> Which version of yt are you using? >> >> >>>> >>>>>>>>>> >> >> >>>> >>>>>>>>>> If you're using 3.0, this is actually fairly easy. >> >> >>>> >>>>>>>>>> If you >> >> >>>> >>>>>>>>>> look in yt.fields.particle_fields.py, around line >> >> >>>> >>>>>>>>>> 85, you >> >> >>>> >>>>>>>>>> can see how this >> >> >>>> >>>>>>>>>> is done for the "particle_density" and >> >> >>>> >>>>>>>>>> "particle_mass" >> >> >>>> >>>>>>>>>> fields. Basically you >> >> >>>> >>>>>>>>>> can call a "deposit" method which takes the >> >> >>>> >>>>>>>>>> particle field >> >> >>>> >>>>>>>>>> quantity you want >> >> >>>> >>>>>>>>>> deposited and deposits it into cells. The >> >> >>>> >>>>>>>>>> underlying >> >> >>>> >>>>>>>>>> calculation is done >> >> >>>> >>>>>>>>>> using Cython, so it's fast. >> >> >>>> >>>>>>>>> >> >> >>>> >>>>>>>>> >> >> >>>> >>>>>>>>> And you shouldn't ever actually need to call these >> >> >>>> >>>>>>>>> "deposit" >> >> >>>> >>>>>>>>> functions, since "deposit" is exposed as a field >> >> >>>> >>>>>>>>> type for >> >> >>>> >>>>>>>>> all datasets that >> >> >>>> >>>>>>>>> contain particles. >> >> >>>> >>>>>>>>> >> >> >>>> >>>>>>>>> Here is a notebook that does this for Enzo AMR >> >> >>>> >>>>>>>>> data: >> >> >>>> >>>>>>>>> >> >> >>>> >>>>>>>>> >> >> >>>> >>>>>>>>> >> >> >>>> >>>>>>>>> >> >> >>>> >>>>>>>>> http://nbviewer.ipython.org/gist/ngoldbaum/5e19e4e6cc2bf330149c >> >> >>>> >>>>>>>>> >> >> >>>> >>>>>>>>> This dataset contains about a million particles and >> >> >>>> >>>>>>>>> generates >> >> >>>> >>>>>>>>> a CIC deposition for the whole domain in about 6 >> >> >>>> >>>>>>>>> seconds >> >> >>>> >>>>>>>>> from a cold start. >> >> >>>> >>>>>>>>> >> >> >>>> >>>>>>>>>> >> >> >>>> >>>>>>>>>> >> >> >>>> >>>>>>>>>> If you're using 2.x, then you can do the same >> >> >>>> >>>>>>>>>> thing, but >> >> >>>> >>>>>>>>>> it's >> >> >>>> >>>>>>>>>> not as straightforward. You can see how this works >> >> >>>> >>>>>>>>>> in >> >> >>>> >>>>>>>>>> yt.data_objects.universal_fields.py, around line >> >> >>>> >>>>>>>>>> 986, >> >> >>>> >>>>>>>>>> where the >> >> >>>> >>>>>>>>>> "particle_density" field is defined. Basically, it >> >> >>>> >>>>>>>>>> calls >> >> >>>> >>>>>>>>>> CICDeposit_3, which >> >> >>>> >>>>>>>>>> is in yt.utilities.lib.CICDeposit.pyx. >> >> >>>> >>>>>>>>>> >> >> >>>> >>>>>>>>>> Let me know if you need any more clarification. >> >> >>>> >>>>>>>>>> >> >> >>>> >>>>>>>>>> Best, >> >> >>>> >>>>>>>>>> >> >> >>>> >>>>>>>>>> John Z >> >> >>>> >>>>>>>>>> >> >> >>>> >>>>>>>>>> On Jun 5, 2014, at 6:07 PM, Brendan Griffen >> >> >>>> >>>>>>>>>> <brendan.f.griffen@gmail.com> wrote: >> >> >>>> >>>>>>>>>> >> >> >>>> >>>>>>>>>> > Hi, >> >> >>>> >>>>>>>>>> > >> >> >>>> >>>>>>>>>> > I was wondering if there were any Cython >> >> >>>> >>>>>>>>>> > routines within >> >> >>>> >>>>>>>>>> > yt >> >> >>>> >>>>>>>>>> > which takes particle data and converts it into a >> >> >>>> >>>>>>>>>> > cloud-in-cell based mesh >> >> >>>> >>>>>>>>>> > which can be written to a file of my choosing. >> >> >>>> >>>>>>>>> >> >> >>>> >>>>>>>>> >> >> >>>> >>>>>>>>> What sort of mesh were you looking for? yt will >> >> >>>> >>>>>>>>> internally >> >> >>>> >>>>>>>>> construct an octree if it is fed particle data. >> >> >>>> >>>>>>>>> I'm not >> >> >>>> >>>>>>>>> sure whether this >> >> >>>> >>>>>>>>> octree can be saved to disk for later analysis. >> >> >>>> >>>>>>>>> >> >> >>>> >>>>>>>>> It's also possible to create a uniform resolution >> >> >>>> >>>>>>>>> covering >> >> >>>> >>>>>>>>> grid containing field data for a deposited >> >> >>>> >>>>>>>>> quantity, which >> >> >>>> >>>>>>>>> can be quite >> >> >>>> >>>>>>>>> easily saved to disk in a number of ways. >> >> >>>> >>>>>>>>> >> >> >>>> >>>>>>>>>> >> >> >>>> >>>>>>>>>> I heard a while ago there was some such >> >> >>>> >>>>>>>>>> functionality but >> >> >>>> >>>>>>>>>> it >> >> >>>> >>>>>>>>>> could be too far down the yt rabbit hole to be >> >> >>>> >>>>>>>>>> used as a >> >> >>>> >>>>>>>>>> standalone? Is this >> >> >>>> >>>>>>>>>> true? I have my own Python code for doing it but >> >> >>>> >>>>>>>>>> it just >> >> >>>> >>>>>>>>>> isn't fast enough >> >> >>>> >>>>>>>>>> and thought I'd ask the yt community if there were >> >> >>>> >>>>>>>>>> any >> >> >>>> >>>>>>>>>> wrapper tools >> >> >>>> >>>>>>>>>> available to boost the speed. >> >> >>>> >>>>>>>>>> > >> >> >>>> >>>>>>>>>> > Thanks. >> >> >>>> >>>>>>>>>> > Brendan >> >> >>>> >>>>>>>>>> >
>> >> >>>> >>>>> YTDomainOverflow Traceback >> >> >>>> >>>>> (most >> >> >>>> >>>>> recent >> >> >>>> >>>>> call last) >> >> >>>> >>>>> >> >> >>>> >>>>> >> >> >>>> >>>>> >> >> >>>> >>>>> /nfs/blank/h4231/bgriffen/data/lib/yt-x86_64/lib/python2.7/site-packages/IPython/utils/py3compat.pyc >> >> >>>> >>>>> in execfile(fname, *where) >> >> >>>> >>>>> 202 else: >> >> >>>> >>>>> 203 filename = fname >> >> >>>> >>>>> --> 204 __builtin__.execfile(filename, >> >> >>>> >>>>> *where) >> >> >>>> >>>>> >> >> >>>> >>>>> >> >> >>>> >>>>> >> >> >>>> >>>>> >> >> >>>> >>>>> /nfs/blank/h4231/bgriffen/work/projects/caterpillar/c2ray/cic/ytcic.py in >> >> >>>> >>>>> <module>() >> >> >>>> >>>>> 52 >> >> >>>> >>>>> 53 ad = ds.all_data() >> >> >>>> >>>>> ---> 54 print ad['deposit', 'all_cic'] >> >> >>>> >>>>> 55 slc = yt.SlicePlot(ds, 2, ('deposit', >> >> >>>> >>>>> 'all_cic')) >> >> >>>> >>>>> 56 slc.set_figure_size(4) >> >> >>>> >>>>> >> >> >>>> >>>>> >> >> >>>> >>>>> >> >> >>>> >>>>> >> >> >>>> >>>>> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc >> >> >>>> >>>>> in __getitem__(self, key) >> >> >>>> >>>>> 205 Returns a single field. Will add if >> >> >>>> >>>>> necessary. >> >> >>>> >>>>> 206 """ >> >> >>>> >>>>> --> 207 f = self._determine_fields([key])[0] >> >> >>>> >>>>> 208 if f not in self.field_data and key not >> >> >>>> >>>>> in >> >> >>>> >>>>> self.field_data: >> >> >>>> >>>>> 209 if f in self._container_fields: >> >> >>>> >>>>> >> >> >>>> >>>>> >> >> >>>> >>>>> >> >> >>>> >>>>> >> >> >>>> >>>>> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.pyc >> >> >>>> >>>>> in _determine_fields(self, fields) >> >> >>>> >>>>> 453 raise >> >> >>>> >>>>> YTFieldNotParseable(field) >> >> >>>> >>>>> 454 ftype, fname = field >> >> >>>> >>>>> --> 455 finfo = >> >> >>>> >>>>> self.pf._get_field_info(ftype, >> >> >>>> >>>>> fname) >> >> >>>> >>>>> 456 else: >> >> >>>> >>>>> 457 fname = field >> >> >>>> >>>>> >> >> >>>> >>>>> >> >> >>>> >>>>> >> >> >>>> >>>>> >> >> >>>> >>>>> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/static_output.pyc >> >> >>>> >>>>> in _get_field_info(self, ftype, fname) >> >> >>>> >>>>> 445 _last_finfo = None >> >> >>>> >>>>> 446 def _get_field_info(self, ftype, fname = >> >> >>>> >>>>> None): >> >> >>>> >>>>> --> 447 self.index >> >> >>>> >>>>> 448 if fname is None: >> >> >>>> >>>>> 449 ftype, fname = "unknown", ftype >> >> >>>> >>>>> >> >> >>>> >>>>> >> >> >>>> >>>>> >> >> >>>> >>>>> >> >> >>>> >>>>> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/data_objects/static_output.pyc >> >> >>>> >>>>> in index(self) >> >> >>>> >>>>> 277 raise RuntimeError("You should >> >> >>>> >>>>> not >> >> >>>> >>>>> instantiate Dataset.") >> >> >>>> >>>>> 278 self._instantiated_index = >> >> >>>> >>>>> self._index_class( >> >> >>>> >>>>> --> 279 self, >> >> >>>> >>>>> dataset_type=self.dataset_type) >> >> >>>> >>>>> 280 # Now we do things that we need an >> >> >>>> >>>>> instantiated index for >> >> >>>> >>>>> 281 # ...first off, we create our >> >> >>>> >>>>> field_info >> >> >>>> >>>>> now. >> >> >>>> >>>>> >> >> >>>> >>>>> >> >> >>>> >>>>> >> >> >>>> >>>>> >> >> >>>> >>>>> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/frontends/stream/data_structures.pyc >> >> >>>> >>>>> in __init__(self, pf, dataset_type) >> >> >>>> >>>>> 942 def __init__(self, pf, dataset_type = >> >> >>>> >>>>> None): >> >> >>>> >>>>> 943 self.stream_handler = pf.stream_handler >> >> >>>> >>>>> --> 944 super(StreamParticleIndex, >> >> >>>> >>>>> self).__init__(pf, >> >> >>>> >>>>> dataset_type) >> >> >>>> >>>>> 945 >> >> >>>> >>>>> 946 def _setup_data_io(self): >> >> >>>> >>>>> >> >> >>>> >>>>> >> >> >>>> >>>>> >> >> >>>> >>>>> >> >> >>>> >>>>> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/geometry/particle_geometry_handler.pyc >> >> >>>> >>>>> in __init__(self, pf, dataset_type) >> >> >>>> >>>>> 48 self.directory = >> >> >>>> >>>>> os.path.dirname(self.index_filename) >> >> >>>> >>>>> 49 self.float_type = np.float64 >> >> >>>> >>>>> ---> 50 super(ParticleIndex, self).__init__(pf, >> >> >>>> >>>>> dataset_type) >> >> >>>> >>>>> 51 >> >> >>>> >>>>> 52 def _setup_geometry(self): >> >> >>>> >>>>> >> >> >>>> >>>>> >> >> >>>> >>>>> >> >> >>>> >>>>> >> >> >>>> >>>>> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/geometry/geometry_handler.pyc >> >> >>>> >>>>> in __init__(self, pf, dataset_type) >> >> >>>> >>>>> 54 >> >> >>>> >>>>> 55 mylog.debug("Setting up domain >> >> >>>> >>>>> geometry.") >> >> >>>> >>>>> ---> 56 self._setup_geometry() >> >> >>>> >>>>> 57 >> >> >>>> >>>>> 58 mylog.debug("Initializing data grid >> >> >>>> >>>>> data IO") >> >> >>>> >>>>> >> >> >>>> >>>>> >> >> >>>> >>>>> >> >> >>>> >>>>> >> >> >>>> >>>>> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/geometry/particle_geometry_handler.pyc >> >> >>>> >>>>> in _setup_geometry(self) >> >> >>>> >>>>> 52 def _setup_geometry(self): >> >> >>>> >>>>> 53 mylog.debug("Initializing Particle >> >> >>>> >>>>> Geometry >> >> >>>> >>>>> Handler.") >> >> >>>> >>>>> ---> 54 self._initialize_particle_handler() >> >> >>>> >>>>> 55 >> >> >>>> >>>>> 56 >> >> >>>> >>>>> >> >> >>>> >>>>> >> >> >>>> >>>>> >> >> >>>> >>>>> >> >> >>>> >>>>> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/geometry/particle_geometry_handler.pyc >> >> >>>> >>>>> in _initialize_particle_handler(self) >> >> >>>> >>>>> 87 pf.domain_left_edge, >> >> >>>> >>>>> pf.domain_right_edge, >> >> >>>> >>>>> 88 [N, N, N], >> >> >>>> >>>>> len(self.data_files)) >> >> >>>> >>>>> ---> 89 self._initialize_indices() >> >> >>>> >>>>> 90 self.oct_handler.finalize() >> >> >>>> >>>>> 91 self.max_level = >> >> >>>> >>>>> self.oct_handler.max_level >> >> >>>> >>>>> >> >> >>>> >>>>> >> >> >>>> >>>>> >> >> >>>> >>>>> >> >> >>>> >>>>> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/geometry/particle_geometry_handler.pyc >> >> >>>> >>>>> in _initialize_indices(self) >> >> >>>> >>>>> 109 npart = >> >> >>>> >>>>> sum(data_file.total_particles.values()) >> >> >>>> >>>>> 110 morton[ind:ind + npart] = \ >> >> >>>> >>>>> --> 111 >> >> >>>> >>>>> self.io._initialize_index(data_file, >> >> >>>> >>>>> self.regions) >> >> >>>> >>>>> 112 ind += npart >> >> >>>> >>>>> 113 morton.sort() >> >> >>>> >>>>> >> >> >>>> >>>>> >> >> >>>> >>>>> >> >> >>>> >>>>> >> >> >>>> >>>>> /bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/frontends/stream/io.pyc in >> >> >>>> >>>>> _initialize_index(self, data_file, regions) >> >> >>>> >>>>> 144 raise >> >> >>>> >>>>> YTDomainOverflow(pos.min(axis=0), >> >> >>>> >>>>> pos.max(axis=0), >> >> >>>> >>>>> 145 >> >> >>>> >>>>> data_file.pf.domain_left_edge, >> >> >>>> >>>>> --> 146 >> >> >>>> >>>>> data_file.pf.domain_right_edge) >> >> >>>> >>>>> 147 regions.add_data_file(pos, >> >> >>>> >>>>> data_file.file_id) >> >> >>>> >>>>> 148 morton.append(compute_morton( >> >> >>>> >>>>> >> >> >>>> >>>>> YTDomainOverflow: Particle bounds [ 0. 0. 0.] and [ >> >> >>>> >>>>> 99.99999237 >> >> >>>> >>>>> 99.99999237 99.99999237] exceed domain bounds [ 0.
>> >> >>>> >>>>>>>>>> > yt-users mailing list >> >> >>>> >>>>>>>>>> > yt-users@lists.spacepope.org >> >> >>>> >>>>>>>>>> > >> >> >>>> >>>>>>>>>> > >> >> >>>> >>>>>>>>>> > >> >> >>>> >>>>>>>>>> > http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >> >> >>>> >>>>>>>>>> >> >> >>>> >>>>>>>>>> _______________________________________________ >> >> >>>> >>>>>>>>>> yt-users mailing list >> >> >>>> >>>>>>>>>> yt-users@lists.spacepope.org >> >> >>>> >>>>>>>>>> >> >> >>>> >>>>>>>>>> >> >> >>>> >>>>>>>>>> >> >> >>>> >>>>>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >> >> >>>> >>>>>>>>> >> >> >>>> >>>>>>>>> >> >> >>>> >>>>>>>>> >> >> >>>> >>>>>>>>> _______________________________________________ >> >> >>>> >>>>>>>>> yt-users mailing list >> >> >>>> >>>>>>>>> yt-users@lists.spacepope.org >> >> >>>> >>>>>>>>> >> >> >>>> >>>>>>>>> >> >> >>>> >>>>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >> >> >>>> >>>>>>>>> >> >> >>>> >>>>>>>> >> >> >>>> >>>>>>>> >> >> >>>> >>>>>>>> _______________________________________________ >> >> >>>> >>>>>>>> yt-users mailing list >> >> >>>> >>>>>>>> yt-users@lists.spacepope.org >> >> >>>> >>>>>>>> >> >> >>>> >>>>>>>> >> >> >>>> >>>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >> >> >>>> >>>>>>>> >> >> >>>> >>>>>>> >> >> >>>> >>>>>>> >> >> >>>> >>>>>>> _______________________________________________ >> >> >>>> >>>>>>> yt-users mailing list >> >> >>>> >>>>>>> yt-users@lists.spacepope.org >> >> >>>> >>>>>>> >> >> >>>> >>>>>>> >> >> >>>> >>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >> >> >>>> >>>>>>> >> >> >>>> >>>>>> >> >> >>>> >>>>>> >> >> >>>> >>>>>> _______________________________________________ >> >> >>>> >>>>>> yt-users mailing list >> >> >>>> >>>>>> yt-users@lists.spacepope.org >> >> >>>> >>>>>> >> >> >>>> >>>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >> >> >>>> >>>>>> >> >> >>>> >>>>> >> >> >>>> >>>>> >> >> >>>> >>>>> _______________________________________________ >> >> >>>> >>>>> yt-users mailing list >> >> >>>> >>>>> yt-users@lists.spacepope.org >> >> >>>> >>>>> >> >> >>>> >>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >> >> >>>> >>>>> >> >> >>>> >>>> >> >> >>>> >>>> >> >> >>>> >>>> _______________________________________________ >> >> >>>> >>>> yt-users mailing list >> >> >>>> >>>> yt-users@lists.spacepope.org >> >> >>>> >>>> >> >> >>>> >>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >> >> >>>> >>>> >> >> >>>> >>> >> >> >>>> >>> >> >> >>>> >>> _______________________________________________ >> >> >>>> >>> yt-users mailing list >> >> >>>> >>> yt-users@lists.spacepope.org >> >> >>>> >>> >> >> >>>> >>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >> >> >>>> >>> >> >> >>>> >> >> >> >>>> >> >> >> >>>> >> _______________________________________________ >> >> >>>> >> yt-users mailing list >> >> >>>> >> yt-users@lists.spacepope.org >> >> >>>> >> >> >> >>>> >> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >> >> >>>> >> >> >> >>>> > >> >> >>>> > >> >> >>>> > _______________________________________________ >> >> >>>> > yt-users mailing list >> >> >>>> > yt-users@lists.spacepope.org >> >> >>>> > >> >> >>>> > http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >> >> >>>> > >> >> >>>> >> >> >>>> >> >> >>>> _______________________________________________ >> >> >>>> yt-users mailing list >> >> >>>> yt-users@lists.spacepope.org >> >> >>>> >> >> >>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >> >> >>>> >> >> >>> >> >> >> >> >> >> >> >> >> _______________________________________________ >> >> >> yt-users mailing list >> >> >> yt-users@lists.spacepope.org >> >> >> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >> >> >> >> >> > >> >> > >> >> > _______________________________________________ >> >> > yt-users mailing list >> >> > yt-users@lists.spacepope.org >> >> > http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >> >> > >> >> _______________________________________________ >> >> yt-users mailing list >> >> yt-users@lists.spacepope.org >> >> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >> > >> > >> > >> > _______________________________________________ >> > yt-users mailing list >> > yt-users@lists.spacepope.org >> > http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >> > >> _______________________________________________ >> yt-users mailing list >> yt-users@lists.spacepope.org >> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org > > > > _______________________________________________ > yt-users mailing list > yt-users@lists.spacepope.org > http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
Hi Brendan, (Snipped most of the rest of the message.) Alright, I've issued a pull request that wraps in a few changes that for me dropped memory usage considerably. https://bitbucket.org/yt_analysis/yt/pull-request/945/reduce-memory-usage-fo... There are other things that need to be done; the most prominent is that we absolutely need to figure out how to substride on octrees. Basically, what we're constrained by is that the way the derived fields work, the size of the data chunk being operated on in spatial mode (i.e., Nzones) is the fundamental size by which we allocate memory during the processing of a field. So if we need five fields to generate the sixth one, we have to have those Nfields resident in memory during the generation. As it stands, the octree system is not flexible enough to do substrides on these, which means we have pages of that size getting allocated, which is where the big increase in memory during the computation of "index","cell_volume" takes place. Fixing this overlaps with moving to forest-of-octrees, so I am going to try to up the priority of that. -Matt On Tue, Jun 10, 2014 at 3:00 PM, Brendan Griffen <brendan.f.griffen@gmail.com> wrote:
Thanks very much Matt! I've improved things at my end as well with my script (non-yt related). Hopefully the confluence of the two changes will enabled a successful run.
Thanks.
Brendan
On Tue, Jun 10, 2014 at 3:57 PM, Matthew Turk <matthewturk@gmail.com> wrote:
Hi Brendan,
I think I've found the problem -- volume is making too many copies. I'm working on a fix.
Thanks very much for your help. I only used ~64GB of memory for 1024^3 particles on a 512^3 mesh (using covering_grid). This essentially cut memory usage in half by my guess. [image: Inline image 1] (I know this could be made to look better!, vmin,vmax etc.) BG On Wed, Jun 11, 2014 at 10:56 AM, Matthew Turk <matthewturk@gmail.com> wrote:
Hi Brendan,
(Snipped most of the rest of the message.)
Alright, I've issued a pull request that wraps in a few changes that for me dropped memory usage considerably.
https://bitbucket.org/yt_analysis/yt/pull-request/945/reduce-memory-usage-fo...
There are other things that need to be done; the most prominent is that we absolutely need to figure out how to substride on octrees. Basically, what we're constrained by is that the way the derived fields work, the size of the data chunk being operated on in spatial mode (i.e., Nzones) is the fundamental size by which we allocate memory during the processing of a field. So if we need five fields to generate the sixth one, we have to have those Nfields resident in memory during the generation. As it stands, the octree system is not flexible enough to do substrides on these, which means we have pages of that size getting allocated, which is where the big increase in memory during the computation of "index","cell_volume" takes place.
Fixing this overlaps with moving to forest-of-octrees, so I am going to try to up the priority of that.
-Matt
Thanks very much Matt! I've improved things at my end as well with my
On Tue, Jun 10, 2014 at 3:00 PM, Brendan Griffen <brendan.f.griffen@gmail.com> wrote: script
(non-yt related). Hopefully the confluence of the two changes will enabled a successful run.
Thanks.
Brendan
On Tue, Jun 10, 2014 at 3:57 PM, Matthew Turk <matthewturk@gmail.com> wrote:
Hi Brendan,
I think I've found the problem -- volume is making too many copies. I'm working on a fix.
yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
I'm very glad everything worked out in the end. I hope this has been a positive experience for you despite the initial speed bumps. We very much appreciate your initial bug report and subsequent active participation in the discussion. Hope you find uses for yt in the future :) Best, Nathan
One last thing. When I run through multiple dimensions [256,512,1024]. It seems to only get the right values for the first two in the list. http://pastebin.com/Q7j2XepF Why would this be? Is there a quantity I have to clear from memory or reset in yt upon each loop? Brendan On Wed, Jun 11, 2014 at 1:00 PM, Nathan Goldbaum <nathan12343@gmail.com> wrote:
I'm very glad everything worked out in the end. I hope this has been a positive experience for you despite the initial speed bumps.
We very much appreciate your initial bug report and subsequent active participation in the discussion.
Hope you find uses for yt in the future :)
Best,
Nathan
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/units/yt_array.pyc in convert_to_units(self, units) 366 367 self.units = new_units --> 368 self *= conversion_factor 369 return self 370
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/units/yt_array.pyc in __imul__(self, other) 667 """ See __mul__. """ 668 oth = sanitize_units_mul(self, other) --> 669 return np.multiply(self, oth, out=self) 670 671 def __div__(self, right_object):
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/units/yt_array.pyc in __array_wrap__(self, out_arr, context) 966 # casting to YTArray avoids creating a YTQuantity with size > 1 967 return YTArray(np.array(out_arr, unit)) --> 968 return ret_class(np.array(out_arr), unit) 969 970
MemoryError:
Nathan, any idea why this is copying? We shouldn't be copying here.
HI Nathan, I just increased n_particles = 1e9 (roughly what I have) for the example you gave me and it does indeed crash due to memory. Line # Mem usage Increment Line Contents ================================================ 6 92.121 MiB 0.000 MiB @profile 7 def test(): 8 92.121 MiB 0.000 MiB n_particles = 1e9 9 10 22980.336 MiB 22888.215 MiB ppx, ppy, ppz = 1e6*np.random.normal(size=[3, n_particles]) 11 12 30609.734 MiB 7629.398 MiB ppm = np.ones(n_particles) 13 14 30609.734 MiB 0.000 MiB data = {'particle_position_x': ppx, 15 30609.734 MiB 0.000 MiB 'particle_position_y': ppy, 16 30609.734 MiB 0.000 MiB 'particle_position_z': ppz, 17 30609.734 MiB 0.000 MiB 'particle_mass': ppm, 18 30609.734 MiB 0.000 MiB 'number_of_particles': n_particles} 19 20 30609.738 MiB 0.004 MiB bbox = 1.1*np.array([[min(ppx), max(ppx)], [min(ppy), max(ppy)], [min(ppy), max(ppy)]]) 21 22 30610.027 MiB 0.289 MiB ds = yt.load_uniform_grid(data, [256, 256, 256], length_unit=parsec, mass_unit=1e8*Msun, bbox=bbox) 23 24 30614.352 MiB 4.324 MiB grid_object = ds.index.grids[0] 25 26 uniform_array = grid_object['deposit', 'all_cic'] 27 28 print uniform_array.max() 29 print uniform_array.shape 30 31 plt.imshow(uniform_array[:,:,128].v) 32 33 plt.savefig('test.png') Traceback (most recent call last): File "/nfs/blank/h4231/bgriffen/data/lib/yt-x86_64/lib/python2.7/runpy.py", line 162, in _run_module_as_main "__main__", fname, loader, pkg_name) File "/nfs/blank/h4231/bgriffen/data/lib/yt-x86_64/lib/python2.7/runpy.py", line 72, in _run_code exec code in run_globals File "/bigbang/data/bgriffen/lib/memory_profiler/lib/python/memory_profiler.py", line 821, in <module> execfile(__file__, ns, ns) File "profilecic.py", line 36, in <module> test() File "/bigbang/data/bgriffen/lib/memory_profiler/lib/python/memory_profiler.py", line 424, in f result = func(*args, **kwds) File "profilecic.py", line 28, in test print uniform_array.max() File "profilecic.py", line 28, in test print uniform_array.max() File "/bigbang/data/bgriffen/lib/memory_profiler/lib/python/memory_profiler.py", line 470, in trace_memory_usage mem = _get_memory(-1) File "/bigbang/data/bgriffen/lib/memory_profiler/lib/python/memory_profiler.py", line 69, in _get_memory stdout=subprocess.PIPE File "/nfs/blank/h4231/bgriffen/data/lib/yt-x86_64/lib/python2.7/subprocess.py", line 709, in __init__ errread, errwrite) File "/nfs/blank/h4231/bgriffen/data/lib/yt-x86_64/lib/python2.7/subprocess.py", line 1222, in _execute_child self.pid = os.fork() OSError: [Errno 12] Cannot allocate memory [yt-x86_64] bigbang% So even the "memory efficient" run can't be run on 1024^3 (ndim = 256) on 128GB machine. Though this because of the way the profiler works. Brendan On Mon, Jun 9, 2014 at 9:20 AM, Matthew Turk <matthewturk@gmail.com> wrote:
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/units/yt_array.pyc in convert_to_units(self, units) 366 367 self.units = new_units --> 368 self *= conversion_factor 369 return self 370
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/units/yt_array.pyc in __imul__(self, other) 667 """ See __mul__. """ 668 oth = sanitize_units_mul(self, other) --> 669 return np.multiply(self, oth, out=self) 670 671 def __div__(self, right_object):
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/units/yt_array.pyc in __array_wrap__(self, out_arr, context) 966 # casting to YTArray avoids creating a YTQuantity with size > 1 967 return YTArray(np.array(out_arr, unit)) --> 968 return ret_class(np.array(out_arr), unit) 969 970
MemoryError:
Nathan, any idea why this is copying? We shouldn't be copying here. _______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
Hi Brendan, Looks like 30 gigs before yt gets used at all, which makes sense if you're allocating 4 fields. I have no idea how memory_profiler works; I always use the yt memory profiler. Without knowing where it crashes *inside* yt (looks like memory_profiler swallows that info) I can't say where it's dying. The call you are making to the cic deposition looks like this internally: pos = data[ptype, coord_name] d = data.deposit(pos, [data[ptype, mass_name]], method = "cic") d = data.apply_units(d, data[ptype, mass_name].units) d /= data["index", "cell_volume"] Internally, there is one allocation call inside the deposit function. I have no idea why it is taking up so much memory, and my suspicion is there are unnecessary copies going on *inside* one of these functions. On Mon, Jun 9, 2014 at 9:42 AM, Brendan Griffen <brendan.f.griffen@gmail.com> wrote:
HI Nathan,
I just increased n_particles = 1e9 (roughly what I have) for the example you gave me and it does indeed crash due to memory.
Line # Mem usage Increment Line Contents ================================================ 6 92.121 MiB 0.000 MiB @profile 7 def test(): 8 92.121 MiB 0.000 MiB n_particles = 1e9 9 10 22980.336 MiB 22888.215 MiB ppx, ppy, ppz = 1e6*np.random.normal(size=[3, n_particles]) 11 12 30609.734 MiB 7629.398 MiB ppm = np.ones(n_particles) 13 14 30609.734 MiB 0.000 MiB data = {'particle_position_x': ppx, 15 30609.734 MiB 0.000 MiB 'particle_position_y': ppy, 16 30609.734 MiB 0.000 MiB 'particle_position_z': ppz, 17 30609.734 MiB 0.000 MiB 'particle_mass': ppm, 18 30609.734 MiB 0.000 MiB 'number_of_particles': n_particles} 19 20 30609.738 MiB 0.004 MiB bbox = 1.1*np.array([[min(ppx), max(ppx)], [min(ppy), max(ppy)], [min(ppy), max(ppy)]]) 21 22 30610.027 MiB 0.289 MiB ds = yt.load_uniform_grid(data, [256, 256, 256], length_unit=parsec, mass_unit=1e8*Msun, bbox=bbox) 23 24 30614.352 MiB 4.324 MiB grid_object = ds.index.grids[0] 25 26 uniform_array = grid_object['deposit', 'all_cic'] 27 28 print uniform_array.max() 29 print uniform_array.shape 30 31 plt.imshow(uniform_array[:,:,128].v) 32 33 plt.savefig('test.png')
Traceback (most recent call last): File "/nfs/blank/h4231/bgriffen/data/lib/yt-x86_64/lib/python2.7/runpy.py", line 162, in _run_module_as_main "__main__", fname, loader, pkg_name) File "/nfs/blank/h4231/bgriffen/data/lib/yt-x86_64/lib/python2.7/runpy.py", line 72, in _run_code exec code in run_globals File "/bigbang/data/bgriffen/lib/memory_profiler/lib/python/memory_profiler.py", line 821, in <module> execfile(__file__, ns, ns) File "profilecic.py", line 36, in <module> test() File "/bigbang/data/bgriffen/lib/memory_profiler/lib/python/memory_profiler.py", line 424, in f result = func(*args, **kwds) File "profilecic.py", line 28, in test print uniform_array.max() File "profilecic.py", line 28, in test print uniform_array.max() File "/bigbang/data/bgriffen/lib/memory_profiler/lib/python/memory_profiler.py", line 470, in trace_memory_usage mem = _get_memory(-1) File "/bigbang/data/bgriffen/lib/memory_profiler/lib/python/memory_profiler.py", line 69, in _get_memory stdout=subprocess.PIPE File "/nfs/blank/h4231/bgriffen/data/lib/yt-x86_64/lib/python2.7/subprocess.py", line 709, in __init__ errread, errwrite) File "/nfs/blank/h4231/bgriffen/data/lib/yt-x86_64/lib/python2.7/subprocess.py", line 1222, in _execute_child self.pid = os.fork() OSError: [Errno 12] Cannot allocate memory [yt-x86_64] bigbang%
So even the "memory efficient" run can't be run on 1024^3 (ndim = 256) on 128GB machine. Though this because of the way the profiler works.
Brendan
On Mon, Jun 9, 2014 at 9:20 AM, Matthew Turk <matthewturk@gmail.com> wrote:
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/units/yt_array.pyc in convert_to_units(self, units) 366 367 self.units = new_units --> 368 self *= conversion_factor 369 return self 370
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/units/yt_array.pyc in __imul__(self, other) 667 """ See __mul__. """ 668 oth = sanitize_units_mul(self, other) --> 669 return np.multiply(self, oth, out=self) 670 671 def __div__(self, right_object):
/bigbang/data/bgriffen/lib/yt-x86_64/src/yt-hg/yt/units/yt_array.pyc in __array_wrap__(self, out_arr, context) 966 # casting to YTArray avoids creating a YTQuantity with size > 1 967 return YTArray(np.array(out_arr, unit)) --> 968 return ret_class(np.array(out_arr), unit) 969 970
MemoryError:
Nathan, any idea why this is copying? We shouldn't be copying here. _______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
participants (5)
-
Brendan Griffen -
Desika Narayanan -
John ZuHone -
Matthew Turk -
Nathan Goldbaum