Re: [yt-users] profile plots combining different datasets

I was saving data just because I don't know how to sum up field values (in this case, "cell_mass") in other way. What I ultimately want to do is to make one phase plot which has "density", "temperature" bin fields, and the "cell_mass" is summed up through all different datasets (like, IsolatedGalaxy/galaxy0030/galaxy0030, IsolatedGalaxy/galaxy0029/galaxy0029, IsolatedGalaxy/galaxy0028/galaxy0028, ...). Is it possible to sum up "prof_2d_ds.data ["cell_mass"]" in the example from different datasets? 2017-11-03 19:05 GMT-04:00 Nathan Goldbaum :

So what's happening is that when you're save_as_dataset at the end of your notebook, you save data with incompatible shapes. In particular, 'density' and 'temperature' have shape (128) while 'cell_mass' has shape (128, 128). We should probably error out in cases like this because the resulting dataset is corrupt - all fields in a single dataset should have the same shape. The error you're seeing happens when yt tries to create an empty array to hold the cell_mass data from your saved dataset by using the shape of one of the bin fields. This leads to an error because the cell_mass field in your saved dataset has a different shape.

One thing that might be immediately helpful is to save your individual profile objects you want to sum, reload those, and then combine the data? You can save profile objects with save_as_dataset:

http://yt-project.org/docs/dev/analyzing/saving_data.html#profiles

Another option would be to eschew using the ytdata frontend entirely and just save the density, temperature, and cell_mass profile fields you want save using e.g. h5py and then construct the plot you want to make using those data.

It's not clear to me what you ultimately want to do with your saved data - maybe if you explain that in a little more detail I can suggest what you might want to do to restructure your workflow.

On Fri, Nov 3, 2017 at 5:53 PM, ｔyuta wrote:

...

I tried on "Isolatedgalaxy" data. I made all the outputs to comments.

https://gist.github.com/Yuta-Tarumi/69ed0428db92553cd4f8ef93 92429036#file-phase-plots-summing-up

2017-11-03 18:29 GMT-04:00 Nathan Goldbaum :

...

So the error is coming from inside yt. Somehow yt is creating an array with shape (128) and trying to fill it with values from an array with shape (128, 128), causing numpy to complain. Unfortunately I'm still not sure what the underlying issue is just from looking at your code and the traceback. It's possible it's a yt bug, it's also possible that it's an issue in your notebook.

Is there any chance you can make a script I can run that triggers the issue? That will make it much easier to understand what's going wrong. You could also just share your full notebook.

If you can, please make use of a dataset from yt-project.org/data in your script. If that's not possible, you can also upload a dataset to the yt curldrop using the "yt upload" command line tool:

$ yt upload /path/to/dataset.tar.gz

and then include the link to the dataset that yt upload will print out once it's finished uploading the data.

-Nathan

On Fri, Nov 3, 2017 at 5:15 PM, ｔyuta wrote:

...

Dear Nathan,

Sorry for the incompleteness. This is the error message from jupyter notebook:

ValueErrorTraceback (most recent call last)<ipython-input-63-383c90a16917> in <module>() 11 fields=["cell_mass"], 12 weight_field=None,---> 13 extrema=dict(density=(1e-30, 1e-20), temperature=(1e1, 1e8)) 14 ) 15 /home/ytarumi/.local/yt/yt/data_objects/profiles.pyc in create_profile(data_source, bin_fields, fields, n_bins, extrema, logs, units, weight_field, accumulation, fractional, deposition) 1093 setattr(obj, "fractional", fractional) 1094 if fields is not None:-> 1095 obj.add_fields([field for field in fields]) 1096 for field in fields: 1097 if fractional: /home/ytarumi/.local/yt/yt/data_objects/profiles.pyc in add_fields(self, fields) 125 citer = self.data_source.chunks([], "io") 126 for chunk in parallel_objects(citer):--> 127 self._bin_chunk(chunk, fields, temp_storage) 128 self._finalize_storage(fields, temp_storage) 129 /home/ytarumi/.local/yt/yt/data_objects/profiles.pyc in _bin_chunk(self, chunk, fields, storage) 565 566 def _bin_chunk(self, chunk, fields, storage):--> 567 rv = self._get_data(chunk, fields) 568 if rv is None: return 569 fdata, wdata, (bf_x, bf_y) = rv /home/ytarumi/.local/yt/yt/data_objects/profiles.pyc in _get_data(self, chunk, fields) 253 for i, field in enumerate(fields): 254 units = chunk.ds.field_info[field].output_units--> 255 arr[:,i] = chunk[field][filter].in_units(units) 256 if self.weight_field is not None: 257 units = chunk.ds.field_info[self.weight_field].output_units ValueError: could not broadcast input array from shape (128,128) into shape (128)

2017-11-03 17:08 GMT-04:00 Nathan Goldbaum :

...

Can you share the full traceback along with the error message you're seeing? You can use gist.github.com to avoid cluttering your e-mail.

On Fri, Nov 3, 2017 at 3:54 PM, ｔyuta wrote:

...

Dear yt-users,

Hi, I want to ask you about phase plot. Some days ago, I asked how to sum up different datasets to make one profile plot, and it worked fine. This time, I tried to do the same thing in phase plot, but it did not work fine. Could you help me find out what the problem is?

(inside for loop): profile_medium = yt.create_profile( data_source=medium_region, bin_fields=["density", "temperature"], fields=["cell_mass"], n_bins=[128, 128], units=dict(), logs=dict(), weight_field=None, extrema=dict(density=(1e-30, 1e-20), temperature=(1e1, 1e8)) ) plot_med = yt.ProfilePlot.from_profiles(profile_medium) profile_med = plot_med.profiles[0] profiles_sum_med = profiles_sum_med + profile_med["cell_mass"]

my_data_med = {"density": profile_med.x, "temperature": profile_med.y, "cell_mass": profiles_sum_med} fake_ds_med = {"current_time": yt.YTQuantity(10, "Myr")} yt.save_as_dataset(fake_ds_med, "phaseplots_med_all.h5", my_data_med)

ds = yt.load("phaseplots_med_all.h5") ad = ds.data

profile = yt.create_profile(ad, ["density", "temperature"], n_bins=[128, 128], fields=["cell_mass"], weight_field=None, extrema=dict(density=(1e-30, 1e-20), temperature=(1e1, 1e8)) )

and I got this error:

could not broadcast input array from shape (128,128) into shape (128)

Sincerely, Y.T.

2017-10-26 16:15 GMT-04:00 ｔyuta :

> Hi Suoqing Ji, > > Thanks! I'll try that. > > 2017-10-26 15:59 GMT-04:00 Suoqing Ji : > >> Hi t yuta, >> >> What you can do is to get the data as numpy array directly from the >> profiles you’ve generated, and sum them up by yourself. You can follow the >> last example at: http://yt-project.org/docs >> /dev/visualizing/plots.html#d-profile-plots >> >> For each dataset, you can have one profile: >> profile = plot.profiles[0] >> >> Then you can sum up profile[‘cell_mass'] over different datasets, >> and plot it against profile.x >> >> Best wishes, >> — >> Suoqing Ji >> Ph.D Candidate >> Department of Physics >> University of California, Santa Barbara >> http://physics.ucsb.edu/~suoqing >> >> On Oct 26, 2017, 12:51 PM -0700, ｔyuta , >> wrote: >> >> Dear yt-users, >> >> Hi, I want to ask you about profile plot. I'm analyzing Enzo data, >> and I'm trying to plot metallicity-cellmass of different 12 datasets into >> one curve. I can plot 12 curves on one image, but that's not what I want. >> Does anyone know some good way, or links to documentation? >> I've attached the 12-curve plot. I want to sum up all the curves to >> get one curve. >> >> Thanks, >> Y.T.[image: 埋め込み画像 2] >> _______________________________________________ >> yt-users mailing list >> yt-users@lists.spacepope.org >> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >> >> >> _______________________________________________ >> yt-users mailing list >> yt-users@lists.spacepope.org >> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >> >> >

_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org

_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org

_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org

_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org

_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org

_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org