Mailman 3 June 2018 - yt-users

Question on Using Parallel Functionality
by Lauren Corlies 11 Jun '18

11 Jun '18

Hi everyone, I'm working on doing an analysis on the ~500 outputs I have from a simulation run so naturally I want to do it in parallel. The data lives on Pleiades where a single node has 32/64 Gb depending on the machine you pick. The general code structure is to take a dataset and compute the fluxes of multiple quantities binned in radius. Because the outputs are large, I'd like to load in 1 dataset per node but then use all 16 cores on the node for the radial flux calculations. To test my code, I'm using 2 smaller outputs of ~4.5 Gb each so they should easily fit on one node but I keep getting memory errors from pleiades. The code does run correctly on my laptop. I'm fairly certain I'm not setting up the code correctly with the different num_proc keywords so that it's trying to do the calculation on a single core instead of half the node. I've posted a paired down example of my code to pastebin that uses two outputs from enzo_comoslogy_plus dataset. The code is named "flux_test_parallel.py" and should run if put inside that dataset directory. The parallel portion of the code is preceded by a line of #'s. ( http://paste.yt-project.org/show/21/) Any advice for how to force the parallel structure to use the machine memory correctly or general pointers for this kind of script would be really appreciated! Thanks! Lauren

3 6

Re: Saving halo profiles as a single dataset
by Nick Gnedin 11 Jun '18

11 Jun '18

Britton, Thank you - I will experiment and report performance. I think requiring the same number of bins in all profiles is a reasonable restriction, even if it increases the output size significantly. If someone is concerned with keeping the output size minimized, they can always write halos within several mass ranges into separate outputs with numbers of bins fixed within a single output but varied between mass ranges, and that can be accomplished with the existing functionality straightforwardly. n On 06/11/2018 11:31 AM, Britton Smith wrote: > Hi Nick, > > Great question, I can see how this could become an issue when halo > catalogs get large. > > I've sketched out a way to do this below that may be slightly hacky, but > will get the job done. > http://paste.yt-project.org/show/24/ > > In the above, I create a new callback that attaches the profiles hanging > off the halo object to the halo catalog itself, then combine them into > single arrays at the end and save using yt.save_as_dataset. This will > work fine if the profiles all have the same number of bins. If not, it > would probably be better to write to an hdf5 file by hand with a single > hdf5 group per profile. That said, HDF5 performance is known to degrade > when the number of groups in a file gets large. In any case, hopefully, > something like the above will work. > > Perhaps in the future, we can make some modifications to the actual code > to do something like this if it seems to work well. > > Britton > > On Wed, Jun 6, 2018 at 4:02 AM Nick Gnedin <gnedin(a)fnal.gov > <mailto:gnedin@fnal.gov>> wrote: > > > I would like to save halo profiles for a large simulation. I understand > that I can use a simple callback like this: > > hc.add_callback("save_profiles", storage="virial_profiles", > output_dir="profiles") > > The problem however is that the call back saves each profiles as a > separate h5 file, and for a large simulation the number of files may be > prohibitive. Instead, I would like to save all of the profiles in a > single fileset, ideally as attributes to halo objects. > > For example, inside my profiling callback I can do the following: > > def hprof_function(halo): > ... > pv = yt.create_profile(...) > setattr(halo,"profs",pv) > > However, hc.create() function does not save all the attributes for halo > objects, so halo.profs would not be saved. > > What would be a proper yt way of storing all the profiles as a single > dataset? > > Thank you, > > n > > _______________________________________________ > yt-users mailing list -- yt-users(a)python.org > <mailto:yt-users@python.org> > To unsubscribe send an email to yt-users-leave(a)python.org > <mailto:yt-users-leave@python.org> >

1 0

Saving halo profiles as a single dataset
by Nick Gnedin 11 Jun '18

11 Jun '18

I would like to save halo profiles for a large simulation. I understand that I can use a simple callback like this: hc.add_callback("save_profiles", storage="virial_profiles", output_dir="profiles") The problem however is that the call back saves each profiles as a separate h5 file, and for a large simulation the number of files may be prohibitive. Instead, I would like to save all of the profiles in a single fileset, ideally as attributes to halo objects. For example, inside my profiling callback I can do the following: def hprof_function(halo): ... pv = yt.create_profile(...) setattr(halo,"profs",pv) However, hc.create() function does not save all the attributes for halo objects, so halo.profs would not be saved. What would be a proper yt way of storing all the profiles as a single dataset? Thank you, n

2 1

Re: changing FoF linking length
by Britton Smith 09 Jun '18

09 Jun '18

Hi Kyungjin, When you create the HaloCatalog object, you can provide the following keyword argument: finder_kwargs={"link": 0.3} "finder_kwargs" is a dictionary of keywords that will be passed to the halo finder when it's run. For the FoF halo finder, the "link" keyword does what you want. Britton On Sat, Jun 9, 2018 at 8:33 PM, Kyungjin Ahn <kjahn(a)chosun.ac.kr> wrote: > Hi, > > When doing halo identification with FoF scheme, how does one put in a > linking length that is not equal to 0.2? I can do fof analysis with default > options, but cannot find how to change the linking length to, say, 0.3. > There is a good reason I need to play with this number so I'd appreciate > any help. > > Thanks! > Kyungjin > > -- > Kyungjin Ahn > Associate Professor > Department of Earth Sciences, Astronomy Division > Chosun University > Tel: 062-230-7340 > +82-62-230-7340 (from abroad) > homepage: www.chosun.ac.kr/kjahn > -- Dr. Britton Smith Assistant Research Scientist San Diego Supercomputer Center University of California, San Diego T: +1 858 822 0936 F: +1 858 534 5117

1 0

changing FoF linking length
by Kyungjin Ahn 09 Jun '18

09 Jun '18

Hi, When doing halo identification with FoF scheme, how does one put in a linking length that is not equal to 0.2? I can do fof analysis with default options, but cannot find how to change the linking length to, say, 0.3. There is a good reason I need to play with this number so I'd appreciate any help. Thanks! Kyungjin -- Kyungjin Ahn Associate Professor Department of Earth Sciences, Astronomy Division Chosun University Tel: 062-230-7340 +82-62-230-7340 (from abroad) homepage: www.chosun.ac.kr/kjahn

1 0

draw_inset_box in annotate_timestamp
by Hill, Alex 08 Jun '18

08 Jun '18

In annotate_timestamp, I am finding that the draw_inset_box option does not behave as I expect. It draws the inset box no matter what; the option simply changes whether there’s a black box around the inset box. Previously, the default was not to display an inset box, and I prefer that. I just switched (mostly) from python 2.7 to 3.6; I think but am not certain that this happened with this switch. I’m running yt 3.4.1. Examples: f = yt.SlicePlot(d, 'z', 'temp') f.annotate_timestamp(draw_inset_box=False) f.show() [cid:9270F490-819C-404A-B1BC-3ED84E4BE27A@T-mobile.com] Changing draw_inset_box to True changes the appearance of the inset box: f = yt.SlicePlot(d, 'z', 'temp') f.annotate_timestamp(draw_inset_box=True) f.show() [cid:4F361896-65D4-4D80-A8C7-E39961512C94@T-mobile.com] By setting alpha to zero, I can get the appearance I want, but I don’t think this is consistent with the documentation. Am I missing something here? f = yt.SlicePlot(d, 'z', 'temp') f.annotate_timestamp(draw_inset_box=True, inset_box_args={'alpha': 0.0}) f.show() [cid:A1C66B4A-1BAF-4C51-8251-FB7661CBEF47@T-mobile.com] Cheers, Alex --------- Alex Hill University of British Columbia and Space Science Institute email: ashill(a)astro.ubc.ca<mailto:ashill@astro.ubc.ca> Based at Dominion Radio Astrophysical Observatory, National Research Council Canada Mailing: PO Box 248, Penticton, BC V2A 6K3, Canada Physical: 717 White Lake Road, Kaleden, BC V0H 1K0, Canada phone: +1 250-497-2356

2 3

Fwd: [Matplotlib-devel] 2018 John Hunter Excellence in Plotting Contest
by Nathan Goldbaum 05 Jun '18

05 Jun '18

---------- Forwarded message --------- From: Nelle Varoquaux <nelle.varoquaux(a)gmail.com> Date: Tue, Jun 5, 2018 at 7:07 PM Subject: [Matplotlib-devel] 2018 John Hunter Excellence in Plotting Contest To: matplotlib development list <matplotlib-devel(a)python.org>, scikit-image Mailing List <scikit-image(a)python.org>, Scikit-learn user and developer mailing list <scikit-learn(a)python.org>, SciPy Users List < scipy-user(a)scipy.org>, Discussion of Numerical Python < numpy-discussion(a)python.org>, <python-list(a)python.org> Hello everyone, Sorry about the cross-posting. There's a couple more days to submit to the John Hunter Excellence in Plotting Competition! If you have any scientific plot worth sharing, submit an entry before June 8th. For more information, see below. Thanks, Nelle In memory of John Hunter, we are pleased to be reviving the SciPy John Hunter Excellence in Plotting Competition for 2018. This open competition aims to highlight the importance of data visualization to scientific progress and showcase the capabilities of open source software. Participants are invited to submit scientific plots to be judged by a panel. The winning entries will be announced and displayed at the conference. John Hunter’s family and NumFocus are graciously sponsoring cash prizes for the winners in the following amounts: - 1st prize: $1000 - 2nd prize: $750 - 3rd prize: $500 - Entries must be submitted by June, 8th to the form at https://goo.gl/forms/7q86zgu5OYUOjODH3. - Winners will be announced at Scipy 2018 in Austin, TX. - Participants do not need to attend the Scipy conference. - Entries may take the definition of “visualization” rather broadly. Entries may be, for example, a traditional printed plot, an interactive visualization for the web, or an animation. - Source code for the plot must be provided, in the form of Python code and/or a Jupyter notebook, along with a rendering of the plot in a widely used format. This may be, for example, PDF for print, standalone HTML and Javascript for an interactive plot, or MPEG-4 for a video. If the original data can not be shared for reasons of size or licensing, "fake" data may be substituted, along with an image of the plot using real data. - Each entry must include a 300-500 word abstract describing the plot and its importance for a general scientific audience. - Entries will be judged on their clarity, innovation and aesthetics, but most importantly for their effectiveness in communicating a real-world problem. Entrants are encouraged to submit plots that were used during the course of research or work, rather than merely being hypothetical. - SciPy reserves the right to display any and all entries, whether prize-winning or not, at the conference, use in any materials or on its website, with attribution to the original author(s). SciPy John Hunter Excellence in Plotting Competition Co-Chairs Thomas Caswell Michael Droettboom Nelle Varoquaux _______________________________________________ Matplotlib-devel mailing list Matplotlib-devel(a)python.org https://mail.python.org/mailman/listinfo/matplotlib-devel

1 0

Question about deposited fields: follow up
by Nick Gnedin 01 Jun '18

01 Jun '18

I also have a follow-up question. My previous question was about the build-in deposited fields. But when I try to do that myself, units seem to be screwed up. ----------------------- import yt import numpy as np def StellarMass2(field,data): ms = data[('STAR','particle_mass')] if('FieldDetector' in str(type(data))): return ms else: return ms.in_units("Msun") d = yt.load("rei10_a0.1001/rei10_a0.1001.art") d.add_field(("STAR","mass2"),function=StellarMass2,sampling_type="particle",units="Msun") d.add_deposited_particle_field(("STAR","mass2"),"nearest",weight_field='particle_ones') xcen = d.arr([5,5,5],"Mpccm/h") s = d.sphere(xcen,(100,"kpc")) sf = ('STAR','mass2') df = ('deposit','STAR_nn_mass2') print(np.amax(s[sf]),np.amax(s[df])) ---------------------------- > 1625.2196373326824 Msun 1625.2196373326824 Msun I.e, the units of the deposited field are that of mass, rather than density. Even worse, if I treat the deposited field as the mass per cell rather than density, it does not add up: print("sum=%9.3e int=%9.3e"%(np.sum(s[sf].in_units("Msun")),np.sum(s[df].in_units("Msun")))) > sum=3.252e+04 int=1.521e+05 So, I am totally lost about what add_deposited_particle_field() function actually does. n

3 2