Mailman 3 November 2013 - yt-users

Re: [yt-users] yt - clump finding
by Alex Bogert 08 Nov '13

08 Nov '13

> Hi Nathan & Matt, > > I am interested in using YT to find clumps in the Bolshoi grid particle > data. So, I tried to test it with Nathan's ENZO data he gave me (DD0025). I > ran into an errror. > > Thanks for your help! > > Alex > > Error: > >> # set up our namespacefrom yt.mods import * from yt.analysis_modules.level_sets.api import * >> fn = "IsolatedGalaxy/galaxy0030/galaxy0030" # parameter file to loadfield = "Density" # this is the field we look for contours over -- we could do >> # this over anything. Other common choices are 'AveragedDensity' >> # and 'Dark_Matter_Density'.step = 2.0 # This is the multiplicative interval between contours. >> pf = load(fn) # load data >> # We want to find clumps over the entire dataset, so we'll just grab the whole# thing! This is a convenience parameter that prepares an object that covers# the whole domain. Note, though, that it will load on demand and not before!data_source = pf.h.disk([0.5, 0.5, 0.5], [0., 0., 1.], >> 8./pf.units['kpc'], 1./pf.units['kpc']) >> # Now we set some sane min/max values between which we want to find contours.# This is how we tell the clump finder what to look for -- it won't look for# contours connected below or above these threshold values.c_min = 10**na.floor(na.log10(data_source[field]).min() )c_max = 10**na.floor(na.log10(data_source[field]).max()+1) >> # keep only clumps with at least 20 cellsfunction = 'self.data[\'%s\'].size > 20' % field >> # Now find get our 'base' clump -- this one just covers the whole domain.master_clump = Clump(data_source, None, field, function=function) >> # This next command accepts our base clump and we say the range between which# we want to contour. It recursively finds clumps within the master clump, at# intervals defined by the step size we feed it. The current value is# *multiplied* by step size, rather than added to it -- so this means if you# want to look in log10 space intervals, you would supply step = 10.0.find_clumps(master_clump, c_min, c_max, step) >> # As it goes, it appends the information about all the sub-clumps to the# master-clump. Among different ways we can examine it, there's a convenience# function for outputting the full hierarchy to a file.f = open('%s_clump_hierarchy.txt' % pf,'w')amods.level_sets.write_clump_hierarchy(master_clump,0,f)f.close() >> # We can also output some handy information, as well.f = open('%s_clumps.txt' % pf,'w')amods.level_sets.write_clumps(master_clump,0,f)f.close() >> # We can traverse the clump hierarchy to get a list of all of the 'leaf' clumpsleaf_clumps = get_lowest_clumps(master_clump) >> # If you'd like to visualize these clumps, a list of clumps can be supplied to# the "clumps" callback on a plot. First, we create a projection plot:prj = ProjectionPlot(pf, 2, field, center='c', width=(20,'kpc')) >> # Next we annotate the plot with contours on the borders of the clumpsprj.annotate_clumps(leaf_clumps) >> # Lastly, we write the plot to disk.prj.save('clumps') >> # We can also save the clump object to disk to read in later so we don't have# to spend a lot of time regenerating the clump objects.pf.h.save_object(master_clump, 'My_clumps') >> # Later, we can read in the clump object like so,master_clump = pf.h.load_object('My_clumps' >> ) >> >> >> -------------------------------------------------------------------------------------------------------------------------------- >> >> yt : [INFO ] 2013-11-07 12:50:45,658 Parameters: current_time = 0.02500008999 >> yt : [INFO ] 2013-11-07 12:50:45,659 Parameters: domain_dimensions = [64 64 64] >> yt : [INFO ] 2013-11-07 12:50:45,660 Parameters: domain_left_edge = [ 0. 0. 0.] >> yt : [INFO ] 2013-11-07 12:50:45,660 Parameters: domain_right_edge = [ 1. 1. 1.] >> yt : [INFO ] 2013-11-07 12:50:45,661 Parameters: cosmological_simulation = 0.0 >> >> ---------------------------------------------------------------------------TypeError Traceback (most recent call last)<ipython-input-10-f290b737b0f4> in <module>() 34 # *multiplied* by step size, rather than added to it -- so this means if you 35 # want to look in log10 space intervals, you would supply step = 10.0.---> 36 find_clumps(master_clump, c_min, c_max, step) 37 38 # As it goes, it appends the information about all the sub-clumps to the >> /usr/local/yt-conda/src/yt-hg/yt/analysis_modules/level_sets/clump_handling.py in find_clumps(clump, min_val, max_val, d_clump) 168 print "Finding clumps: min: %e, max: %e, step: %f" % (min_val, max_val, d_clump) 169 if min_val >= max_val: return--> 170 clump.find_children(min_val) 171 172 if (len(clump.children) == 1): >> /usr/local/yt-conda/src/yt-hg/yt/analysis_modules/level_sets/clump_handling.py in find_children(self, min_val, max_val) 109 if max_val is None: max_val = self.max_val 110 contour_info = identify_contours(self.data, self.field, min_val, max_val,--> 111 self.cached_fields) 112 for cid in contour_info: 113 new_clump = self.data.extract_region(contour_info[cid]) >> /usr/local/yt-conda/src/yt-hg/yt/analysis_modules/level_sets/contour_finder.py in identify_contours(data_source, field, min_val, max_val, cached_fields) 59 def identify_contours(data_source, field, min_val, max_val, 60 cached_fields=None):---> 61 cur_max_id = np.sum([g.ActiveDimensions.prod() for g in data_source._grids]) 62 pbar = get_pbar("First pass", len(data_source._grids)) 63 grids = sorted(data_source._grids, key=lambda g: -g.Level) >> TypeError: 'NoneType' object is not iterable >> >> >> Finding clumps: min: 1.000000e-30, max: 1.000000e-21, step: 2.000000 >> >> >> >> >> On Thu, Nov 7, 2013 at 12:49 PM, Alex Bogert <bogart.alex(a)gmail.com>wrote: >> >>> >>> >>> ---------- Forwarded message ---------- >>> From: Alex Bogert <bogart.alex(a)gmail.com> >>> Date: Thu, Nov 7, 2013 at 12:46 PM >>> Subject: yt - clump finding >>> To: "Conor Kaminer (Google Drive)" <ckaminer(a)ucsc.edu> >>> >>> >>> # set up our namespacefrom yt.mods import * from yt.analysis_modules.level_sets.api import * >>> fn = "IsolatedGalaxy/galaxy0030/galaxy0030" # parameter file to loadfield = "Density" # this is the field we look for contours over -- we could do >>> # this over anything. Other common choices are 'AveragedDensity' >>> # and 'Dark_Matter_Density'.step = 2.0 # This is the multiplicative interval between contours. >>> pf = load(fn) # load data >>> # We want to find clumps over the entire dataset, so we'll just grab the whole# thing! This is a convenience parameter that prepares an object that covers# the whole domain. Note, though, that it will load on demand and not before!data_source = pf.h.disk([0.5, 0.5, 0.5], [0., 0., 1.], >>> 8./pf.units['kpc'], 1./pf.units['kpc']) >>> # Now we set some sane min/max values between which we want to find contours.# This is how we tell the clump finder what to look for -- it won't look for# contours connected below or above these threshold values.c_min = 10**na.floor(na.log10(data_source[field]).min() )c_max = 10**na.floor(na.log10(data_source[field]).max()+1) >>> # keep only clumps with at least 20 cellsfunction = 'self.data[\'%s\'].size > 20' % field >>> # Now find get our 'base' clump -- this one just covers the whole domain.master_clump = Clump(data_source, None, field, function=function) >>> # This next command accepts our base clump and we say the range between which# we want to contour. It recursively finds clumps within the master clump, at# intervals defined by the step size we feed it. The current value is# *multiplied* by step size, rather than added to it -- so this means if you# want to look in log10 space intervals, you would supply step = 10.0.find_clumps(master_clump, c_min, c_max, step) >>> # As it goes, it appends the information about all the sub-clumps to the# master-clump. Among different ways we can examine it, there's a convenience# function for outputting the full hierarchy to a file.f = open('%s_clump_hierarchy.txt' % pf,'w')amods.level_sets.write_clump_hierarchy(master_clump,0,f)f.close() >>> # We can also output some handy information, as well.f = open('%s_clumps.txt' % pf,'w')amods.level_sets.write_clumps(master_clump,0,f)f.close() >>> # We can traverse the clump hierarchy to get a list of all of the 'leaf' clumpsleaf_clumps = get_lowest_clumps(master_clump) >>> # If you'd like to visualize these clumps, a list of clumps can be supplied to# the "clumps" callback on a plot. First, we create a projection plot:prj = ProjectionPlot(pf, 2, field, center='c', width=(20,'kpc')) >>> # Next we annotate the plot with contours on the borders of the clumpsprj.annotate_clumps(leaf_clumps) >>> # Lastly, we write the plot to disk.prj.save('clumps') >>> # We can also save the clump object to disk to read in later so we don't have# to spend a lot of time regenerating the clump objects.pf.h.save_object(master_clump, 'My_clumps') >>> # Later, we can read in the clump object like so,master_clump = pf.h.load_object('My_clumps') >>> >>> >>> >> >

3 2

EPS.single_plot() interface
by John Regan 06 Nov '13

06 Nov '13

Hi All, I've been playing with the eps interface a little to decorate some plots. It's a very nice feature but I haven't been able to get in to work with the output from OffAxisProjectionPlot() or off_axis_projection(). Basically I call one of these functions, produce a nice projection and would like to pass that object to eps.single_plot and make some modifications. The problem is that PWViewrMPL object has no colorbar attribute and so when insert_image_yt() gets called the script fails as it tries to access this attribute. I can workaround the issue by creating the projection and converting the image to jpg and passing the jpg image to insert_image() and it works great but it's not very clean. It would be nice to pass the PWViewrMPL object in somehow. Just wondering if another wrapper function is needed or if there is an even easier way that I am missing. Cheers, John

2 11

object.hierarchy.ray( )
by Yong Zheng 06 Nov '13

06 Nov '13

Hi, I am trying to use the .hierarchy.ray( ) to extract data along a particular light ray, but when I check the temperature field of that light ray data, I find that the data points it extracted were not really what I want, the temperature range goes beyond my limit, here is part of my code: sp = pf.h.sphere(center, radius) warmhot = sp.cut_region(['grid["Temperature"] >= 1e5', 'grid["Temperature"] < 1e6']) p0 = np.array([center[0], center[1]-250./pf['kpc'], center[2]]) p1 = np.array([center[0], center[1]+250./pf['kpc'], center[2]]) ray = warmhot.hierarchy.ray(p0, p1) tempt = warmhot['Temperature'] ray_tempt = ray['Temperature'] when I examine the temperature range for warmnot, it is (100000.1171875, 999999.5625) but for ray_tempt, it is: (6607.2802734375, 86239664.0) which is strange since the 'ray' data should be a subset of 'warmhot'... Is there any explanation for it? Thanks! - Yong

2 1

Accessing data
by Stuart Mumford 06 Nov '13

06 Nov '13

Hello, I am starting upon an adventure of actually using yt and I am already getting the impression I may be using a chainsaw to cut kindling. My data is uniform grid, not that large (128^3 for now) and I need to access the fields, both on disk and derived in 3D array format. I have tried using a covering_grid thus: ``` ts = yt.load("/archive/io_testing/3D_tube128_Slog_p90_A20r2-3_B005/3D_tube128_Slog_p90_A20r2-3_B005_*.gdf") ds = ts[10] cg = ds.h.covering_grid(0, ds.domain_left_edge, ds.domain_dimensions) cg['density_pert'] ``` which soemtimes seems to give me an array full of -999 and sometimes gives me this lovely traceback: ``` --------------------------------------------------------------------------- DataCubeError Traceback (most recent call last) <ipython-input-55-62e42995fe53> in <module>() ----> 1 cg['density_pert'] /home/stuart/BitBucket/yt/yt/data_objects/data_containers.pyc in __getitem__(self, key) 316 if key not in self.fields: 317 self.fields.append(key) --> 318 self.get_data(key) 319 return self.field_data[key] 320 /home/stuart/BitBucket/yt/yt/data_objects/data_containers.pyc in get_data(self, fields) 3757 for i, grid in enumerate(self._grids): 3758 if self._use_pbar: pbar.update(i) -> 3759 count -= self._get_data_from_grid(grid, obtain_fields) 3760 if count <= 0: break 3761 if self._use_pbar: pbar.finish() /home/stuart/BitBucket/yt/yt/data_objects/data_containers.pyc in save_state(self, grid, field, *args, **kwargs) 79 old_keys = grid.field_data.keys() 80 grid.field_parameters = self.field_parameters ---> 81 tr = func(self, grid, field, *args, **kwargs) 82 grid.field_parameters = old_params 83 grid.field_data = YTFieldData( [(k, grid.field_data[k]) for k in old_keys] ) /home/stuart/BitBucket/yt/yt/data_objects/data_containers.pyc in _get_data_from_grid(self, grid, fields) 3801 c_fields, g_fields, 3802 self.ActiveDimensions, grid.ActiveDimensions, -> 3803 grid.child_mask, self.domain_width, ll, 0) 3804 return count 3805 DataCubeError: DataCubeGeneric: Invalid parameters. ``` I only want my arrays :'( On another, related, note: I have a bit of code that executes in serial at the top of a otherwise embarrassingly parallel piece of vtk code, I need the most optimal io possible for this, as it is all io limited, how well is yt suited to getting one slice out of my gdf files as fast as humanly possible? Stuart

2 6

Setting Color map and x and y axis ticks in PlotCollection or ProjectionPlot
by k.grisdale＠surrey.ac.uk 06 Nov '13

06 Nov '13

Hi Everyone, I am trying to plot 3D data from RAMSES as a projection. The plots I have created with the ProjectionPlot look good (with the command below) but I want to rescale the colour map so that when I compare different times the scale matches. I would also like to change colour map used. [ProjectionPlot(p6,'z','Density',center=[0.5,0.5,0.5]).save(“p6”)] I started by trying to change just the colour map on these plots using the following: prj=ProjectionPlot(p6,'z','Density',center=[0.5,0.5,0.5]).save(“p6”) prj.set_cmap(‘YlOrRd’) but I get the error: Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/Applications/Code/yt/yt-x86_64/src/yt-hg/yt/visualization/plot_window.py", line 107, in newfunc rv = f(*args, **kwargs) and I’m not sure what I am missing. I have now tried a different approach of cen = [0.5,0.5,0.5] pc = PlotCollection(p6, cen) pc.add_projection("Density",2) pc.set_width(0.5, 'cm') pc.set_xlim(-0.35, 0.35) pc.set_ylim(-0.35, 0.35) pc.set_zlim(0.06, 0.22) pc.set_cmap('YlOrRd') pc.save('test’) While this new approach changes the colour map on the image it does update the colour map on the colour bar(not sure if thats the correct name) on the left of the image.The scale however does update. Also this method produces graphs with out ticks for x and y axis which I would like have. Can anyone point out what I am missing or what I can do differently? Thanks In advance Kearn

2 4

Problem trying to define a new field
by k.grisdale＠surrey.ac.uk 06 Nov '13

06 Nov '13

Hi yt-users I’m attempting to plot some profiles as a function of radius/radiusSH, where radiusSh is the radius of a shock. However I am not sure what the simplest way of getting yt to plot this. I have been using the following commands to produce the profiles. : pc = PlotCollection(p6, center = cen) pc.add_profile_sphere(0.5, "cm", ["Radius", "Density"], weight = "Denin”) I have also tried to define a new field: @derived_field(name = "RRsh", take_log = False, validators = [ValidateParameter("rsh")]) def RRshunits(field, data): value, rsh = p6.h.find_max("Density") return(data["Radius"]/rsh) but this leads to the following error: Traceback (most recent call last): File "profiles.py", line 21, in <module> pc.add_profile_sphere(0.5, "cm", ["RRsh", "Denin"], weight = "Denin") File "/Applications/Code/yt/yt-x86_64/src/yt-hg/yt/visualization/plot_collection.py", line 1058, in add_profile_sphere figure=figure, axes=axes) File "/Applications/Code/yt/yt-x86_64/src/yt-hg/yt/visualization/plot_collection.py", line 961, in add_profile_object lazy_reader=lazy_reader)[0] File "/Applications/Code/yt/yt-x86_64/src/yt-hg/yt/data_objects/derived_quantities.py", line 92, in __call__ return self._call_func_lazy(args, kwargs) File "/Applications/Code/yt/yt-x86_64/src/yt-hg/yt/data_objects/derived_quantities.py", line 99, in _call_func_lazy rv = self.func(GridChildMaskWrapper(g, self._data_source), *args, **kwargs) File "/Applications/Code/yt/yt-x86_64/src/yt-hg/yt/data_objects/derived_quantities.py", line 615, in _Extrema if data[field].size < 1: File "/Applications/Code/yt/yt-x86_64/src/yt-hg/yt/data_objects/derived_quantities.py", line 60, in __getitem__ data = self.data_source._get_data_from_grid(self.grid, item) File "/Applications/Code/yt/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.py", line 95, in save_state tr = func(self, grid, field, *args, **kwargs) File "/Applications/Code/yt/yt-x86_64/src/yt-hg/yt/data_objects/data_containers.py", line 2645, in _get_data_from_grid tr = grid[field] File "/Applications/Code/yt/yt-x86_64/src/yt-hg/yt/data_objects/grid_patch.py", line 157, in __getitem__ self.get_data(key) File "/Applications/Code/yt/yt-x86_64/src/yt-hg/yt/data_objects/grid_patch.py", line 200, in get_data self._generate_field(field) File "/Applications/Code/yt/yt-x86_64/src/yt-hg/yt/data_objects/grid_patch.py", line 132, in _generate_field self.pf.field_info[field].check_available(self) File "/Applications/Code/yt/yt-x86_64/src/yt-hg/yt/data_objects/field_info_container.py", line 366, in check_available validator(data) File "/Applications/Code/yt/yt-x86_64/src/yt-hg/yt/data_objects/field_info_container.py", line 442, in __call__ raise NeedsParameter(doesnt_have) yt.data_objects.field_info_container.NeedsParameter: (['rsh’]) Am I missing something in my field definition or is their a simpler way to go about this? Thanks Kearn

2 6

grid levels in "complex" multi-plots
by Wolfram Schmidt 06 Nov '13

06 Nov '13

Hi, does anyone have an idea how to overplot grid levels (as contour lines at the boundaries of refinement levels) in a multi-plot of the type http://yt-project.org/docs/2.5/cookbook/complex_plots.html#multi-plot-slice… Cheers, Wolfram

2 1

getting more information about memory usage
by nick moeckel 06 Nov '13

06 Nov '13

Hi there, I'm trying to analyse some data with a 1024^3 base grid, and it's pushing at the memory limits of the machine where the data resides. So two quick questions: should I expect parallel analysis to have a reduced memory footprint? E.g. if I need 16Gb of memory to run this analysis serially, will running on 16 cores get me anywhere near 1Gb/core? does yt have a more verbose debug mode or something that I can activate to output any details about memory consumption that might aid me in the delicate and adversarial dance that this cluster and I are engaged in? thanks, Nick

2 2

Plotting data from RAMSES
by k.grisdale＠surrey.ac.uk 04 Nov '13

04 Nov '13

Hi yt-users I am trying to use yt to plot some Ramses data. When I use the following the commands I get the output from the first attachment: >>> p6 = load("output_00006/info_00006.txt”) >>> SlicePlot(p6, 'x', 'Density').save() [cid:D7418A7C-97AF-4411-9BD5-844D5E069FD3@surrey.ac.uk] But if I use yt serve to generate the plot with the same data I get this graph: [cid:EA21D51E-3561-4D91-B38F-8A049AD645D7@surrey.ac.uk] Any suggestions on where I’m going wrong or what I am missing? Thanks Kearn

2 2

parallel issue
by nick moeckel 04 Nov '13

04 Nov '13

Hi everyone, I'm trying to run some parallel analysis scripts and I'm running into problems. I'm using yt-3.0 on Ramses data; here's a simple example that doesn't work: from yt.pmods import * ds = load('output_00001/info_00001.txt', fields=['Density','x-velocity','y-velocity','z-velocity','Pressure']) proj = ProjectionPlot(ds, 0, 'Density') proj.save() the errors this generates: http://paste.yt-project.org/show/SfT87oaHcQpYMUnBi2QM/ When I run this on my personal computers with mpi4py installed with pip, this will work with no problems. On the HPC system I'm using I'm unable to install mpi4py with pip, (due [I think] to the archaic openssl version) but I've built and installed it from source. Before I tear any more of my hair out over this, I'm hoping that someone can tell me if this looks familiar, or if it's definitely a problem with mpi4py and yt playing nice, or anything else that might save me a few days of my life. Thanks for any help! Nick

3 4