yt-dev November 2009

yt-dev@python.org

6 participants
12 discussions

nickel instruction for using parallel HOP (not on Kraken)
by Stephen Skory 01 Dec '09

01 Dec '09

Hi all, Matt asked me to outline how to get parallel HOP (in the hg devel version) working. In easy to follow list format: 0. Build yt like you did already (you're on this list, no?). 1. Get the latest hg, choose the 'yt' branch, which is the default. 2. Install Forthon in the standard python way. http://hifweb.lbl.gov/Forthon/ 3. Go into yt-devel/yt/extensions/kdtree and I think 'make' will work on most machines. If not, let me know. It should make 'fKDpy.so' in that same dir. 4. Go to the top level of yt-devel, and type 'python setup.py install' like usual, but then do 'cp -r yt/extensions/kdtree ../../lib/python2.6/site-packages/yt-version-num-sys-etc/yt/extensions/'. I haven't added the kdtree stuff to setup utils yet, so this step isn't automated. 5. One runs parallel HOP very similarly to old HOP: h = yt.lagos.HaloFinding.parallelHF(pf, threshold=160.0, safety=1.5, \ dm_only=False,resize=True, fancy_padding=True, rearrange=True) - safety - is a padding multiplier that adjusts the size of the padding around each subvolume. 1.5 is usually sufficient, and 1.35 probably is. There's no reason to try going lower from 1.5 unless memory is a concern. - dm_only - I currently have things a little messed up in HaloFinding.py in the dev, so this needs to be set unless you have a ['creation_time'] field. - resize - Load balancing. This is a recursive bisection that subdivides the domains like a kd tree, but it can do arbitrary numbers of cuts, so you can run with any number of tasks. Generally, I turn it on for datasets smaller than 300 Mpc, and for ones larger, the particles are already pretty evenly balanced and it doesn't help. - fancy_padding - Each subvolume has six different faces, and six different values for the padding, one for each face. There is really no reason to turn this off. - rearrange - The fortran kdtree can make an internal copy of the point data which increases the speed of the searches by almost 20%. It however increases the memory by 3 fields, so if memory is a concern, turn this off. 6. The usual halo convenience functions work as well, but the output is a bit different. h.write_out('dist-chain.out') - Just like the normal one. h.write_particle_lists("chain") - One hdf5 file per task, that contains the particle data for all the haloes on that task. h.write_particle_lists_txt("chain") - One text file that lists which tasks own a piece of a halo. Since haloes can exist on more than one task, each halo may have several entries on it's line. 7. There is a rough memory scaling law: it takes 1 MB of RAM per 4,500 particles, with rearrange turned on, regardless of task count. This does not include the yt hierarchy costs, so if you have a dataset like L7, which uses 1GB per task, you will need to take that into account, too. Good luck! _______________________________________________________ sskory(a)physics.ucsd.edu o__ Stephen Skory http://physics.ucsd.edu/~sskory/ _.>/ _Graduate Student ________________________________(_)_\(_)_______________

3 10

particle io issue
by Britton Smith 28 Nov '09

28 Nov '09

Hey everyone, The following error pertains to the yt branch of the hg repo. I'm getting a particle io error when using the various yt halo finders in parallel. It doesn't appear to have anything to do with the halo finders, but I don't know what else uses ParticleIO.py. This only happens when running in parallel. P000 yt INFO 2009-11-27 15:16:59,775 Getting ParticleMassMsun using ParticleIO Setting period equal to 1.000000 Setting period equal to 1.000000 Setting period equal to 1.000000 Traceback (most recent call last): File "do_hop.py", line 5, in <module> h = FOFHaloFinder(pf) File "/Users/britton/Documents/work/yt-hg/yt/lagos/HaloFinding.py", line 1024, in __init__ self._parse_halolist(1.) File "/Users/britton/Documents/work/yt-hg/yt/lagos/HaloFinding.py", line 747, in _parse_halolist this_max_dens = halo.maximum_density_location() File "/Users/britton/Documents/work/yt-hg/yt/lagos/ParallelTools.py", line 130, in single_proc_results return func(self, *args, **kwargs) File "/Users/britton/Documents/work/yt-hg/yt/lagos/HaloFinding.py", line 322, in maximum_density_location return self.center_of_mass() File "/Users/britton/Documents/work/yt-hg/yt/lagos/ParallelTools.py", line 130, in single_proc_results return func(self, *args, **kwargs) File "/Users/britton/Documents/work/yt-hg/yt/lagos/HaloFinding.py", line 306, in center_of_mass pm = self["ParticleMassMsun"] File "/Users/britton/Documents/work/yt-hg/yt/lagos/ParallelTools.py", line 130, in single_proc_results return func(self, *args, **kwargs) File "/Users/britton/Documents/work/yt-hg/yt/lagos/HaloFinding.py", line 136, in __getitem__ return self.data.particles[key][self.indices] File "/Users/britton/Documents/work/yt-hg/yt/lagos/ParticleIO.py", line 46, in __getitem__ self.get_data(key) File "/Users/britton/Documents/work/yt-hg/yt/lagos/ParticleIO.py", line 104, in get_data if len(to_add) != 1: raise KeyError KeyError I checked the contents of to_add, and it was a list with two items (hence the exception): ['particle_mass', 'particle_mass'] I was able to get around this by removing non-unique entries in to_add, but I don't think that really fixes the underlying problem. Anyone have any ideas? Regards, Britton

2 2

Cython 0.12
by Matthew Turk 23 Nov '09

23 Nov '09

Cython 0.12 has been released: http://pypi.python.org/pypi/Cython/0.12 The release notes: http://wiki.cython.org/ReleaseNotes-0.12 are interesting, as they indicate type inference is now available, which should make writing fast extensions much easier. C pointers can also now be sliced and iterated over. -Matt

1 0

Changes to the trunk
by Matthew Turk 20 Nov '09

20 Nov '09

Hi there, Tonight I'm merging a bunch of changes from me, Britton Smith, John Wise and Stephen Skory back into the trunk from the mercurial repository. If you're running on the yt-1.5 source tree, this won't affect you -- but if you're running on trunk, the next time you update you will have to rerun "python2.6 setup.py develop" or "python2.6 setup.py install" depending on how you installed. If you used the install script, you should also be able to re-run it at any time to get an updated source tree and installation. There are no pressing fixes in this, only speed and memory improvements, so don't feel it's necessary to make the jump just now if you're happy with where you are. If you do update, here's a minor changelog: * Hierarchy is much faster, and the .yt file now only stores data products, all hierarchy info goes into a .harrays file (for forward compatibility with Enzo) * The attributes off the hierarchy for grids: gridLeftEdge and gridRightEdge are now grid_left_edge and grid_right_edge, in keeping with our coding standards * Hierarchy has been rewritten for *clarity* * The bug with DM-only sims has been fixed * IO routines have been refactored and very cleaned up * Preliminary particle IO readers have been added -- much, much faster, but not yet production-ready * New cloud-in-cell deposit function without needing fortran * Some other minor things to support coming features All the public and primary functions, classes etc should be the same. Thanks, and let us know if there are ANY problems! -Matt

1 0

Packed AMR hierarchy problem
by Stephen Skory 19 Nov '09

19 Nov '09

Hi (Matt, mainly), I have a packed AMR, dm-only dataset that is giving me some odd issues with the hierarchy. The grids are not being referenced to the correct AMR HDF5 file. These commands: from yt.mods import * pf = load('RedshiftOutput0005', data_style="enzo_packed_3d") pf.h.cpu_map['/mirage/sskory/reddead-survey-amr/RD0005/RedshiftOutput0005.cpu0004'][:10] give: [EnzoGrid_0005, EnzoGrid_1867, EnzoGrid_1868, EnzoGrid_1869, EnzoGrid_1870, EnzoGrid_1871, EnzoGrid_1872, EnzoGrid_1873, EnzoGrid_1874, EnzoGrid_1875] This shows that grid 1867 is owned by cpu 4. But here's what the .hierarchy file says about Grid 1867, that it's owned by cpu0005. A manual inspection shows that it is indeed inside of the cpu0005 HDF5 file. Grid = 1867 Task = 5 GridRank = 3 GridDimension = 12 12 12 GridStartIndex = 3 3 3 GridEndIndex = 8 8 8 GridLeftEdge = 0.5078125 0.521484375 0.337890625 GridRightEdge = 0.513671875 0.52734375 0.34375 Time = 646.75066015177 SubgridsAreStatic = 0 NumberOfBaryonFields = 0 NumberOfParticles = 96 ParticleFileName = /mirage/sskory/reddead-survey-amr/RD0005/RedshiftOutput0005.cpu0005 GravityBoundaryType = 2 Pointer: Grid[1867]->NextGridThisLevel = 0 Pointer: Grid[1867]->NextGridNextLevel = 0 Pointer: Grid[5]->NextGridNextLevel = 1868 I'm using the latest trunk, and the files should be world-readable. Any ideas why this is happening? Thanks! _______________________________________________________ sskory(a)physics.ucsd.edu o__ Stephen Skory http://physics.ucsd.edu/~sskory/ _.>/ _Graduate Student ________________________________(_)_\(_)_______________

3 8

PyPy benchmarks
by Matthew Turk 18 Nov '09

18 Nov '09

Hi all, I saw this post today: http://morepypy.blogspot.com/2009/11/some-benchmarking.html It's about benchmarks for the PyPy project, with a just-in-time compiler enabled; PyPy is a reimplementation of Python in a restricted version of Python, which aims to be far more flexible as well as faster. (Also, some other language-engineer goals I don't really know much about!) Very impressive, but I'm pretty sure it's not yet a target for deployment -- but maybe some day, and it's becoming clear that Python's speed will improve with time. -Matt

1 0

paste extension
by Matthew Turk 16 Nov '09

16 Nov '09

Hi guys, There's a PasteBin extension that's compatible with paste.enzotools.org: http://mercurial.selenic.com/wiki/PasteExtension You just have to set the server to http://paste.enzotools.org/ to use it. There's docs at the top of the file. -Matt

2 1

Fwd: [Yt-svn] yt-commit r1527 - trunk/yt/lagos
by Matthew Turk 16 Nov '09

16 Nov '09

Just as a note, I noticed a problem with the unit tests today and traced it back to a bug I introduced in the profiling code about two weeks ago. It's now fixed, but please be sure to update trunk to r1527. ---------- Forwarded message ---------- From: <mturk(a)wrangler.dreamhost.com> Date: Mon, Nov 16, 2009 at 1:16 PM Subject: [Yt-svn] yt-commit r1527 - trunk/yt/lagos To: yt-svn(a)lists.spacepope.org Author: mturk Date: Mon Nov 16 13:16:27 2009 New Revision: 1527 URL: http://yt.enzotools.org/changeset/1527 Log: This fixes a regression in the lazy profiling! For a very brief window of time, child cells were not being masked in the profiling method. Modified: trunk/yt/lagos/Profiles.py Modified: trunk/yt/lagos/Profiles.py ============================================================================== --- trunk/yt/lagos/Profiles.py (original) +++ trunk/yt/lagos/Profiles.py Mon Nov 16 13:16:27 2009 @@ -160,9 +160,14 @@ data = [] for field in _field_mapping.get(this_field, (this_field,)): pointI = None - if check_cut and not self._data_source._is_fully_enclosed(source): + if check_cut: + # This conditional is so that we can have variable-length + # particle fields. Note that we can't apply the + # is_fully_enclosed to baryon fields, because child cells get + # in the way. if field in self.pf.field_info \ - and self.pf.field_info[field].particle_type: + and self.pf.field_info[field].particle_type \ + and not self._data_source._is_fully_enclosed(source): pointI = self._data_source._get_particle_indices(source) else: pointI = self._data_source._get_point_indices(source) _______________________________________________ Yt-svn mailing list Yt-svn(a)lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-svn-spacepope.org

1 0

Projection speed improvement patch
by Matthew Turk 09 Nov '09

09 Nov '09

Hi guys, (For all of these performance indicators, I've used the 512^3 L7 amr-everywhere run called the "LightCone." This particular dataset has ~380,000 grids and is a great place to find the ) Last weekend I did a little bit of benchmarking and saw that the parallel projections (and likely several other parallel operations) all sat inside an MPI_Barrier for far too long. I converted (I think!) this process to be an MPI_Alltoallv operation, following on an MPI_Allreduce to get the final array size and the offsets into an ordered array, and I think it is working. I saw pretty good performance improvements, but it's tough to quantify those right now -- for projecting "Ones" (no disk-access) it sped things up by ~15%. I've also added a new binary hierarchy method to devel enzo, and it provides everything that is necessary for yt to analyze the data. As such, if a %(basename)s.harrays file exists, it will be used, and yt will not need to open the .hierarchy file at all. This sped things up by 100 seconds. I've written a script to create these (http://www.slac.stanford.edu/~mturk/create_harrays.py), but outputting them inline in Enzo is the fastest. To top this all off, I ran a projection -- start to finish, including all overhead -- on 16 processors. To project the fields "Density" (native), "Temperature" (native) and "VelocityMagnitude" (derived, requires x-, y- and z-velocity) on 16 processors to the finest resolution (adaptive projection -- to L7) takes 140 seconds, or roughly 2:20. I've looked at the profiling outputs, and it seems to me that there are still some places performance could be squeezed out. That being said, I'm pretty pleased with these results. These are all in the named branch hierarchy-opt in mercurial. They rely on some rearrangement of the hierarchy parsing and whatnot that has lived in hg for a little while; it will go into the trunk as soon as I get the all clear about moving to a proper stable/less-stable dev environment. I also have some other test suites to run on them, and I want to make sure the memory usage is not excessive. Best, Matt

4 16

mpi4py on an altix
by John Wise 08 Nov '09

08 Nov '09

Hi, This is particularly aimed at Matt since he has experience with installing mpi4py on an altix :-) I saw your post on the mpi4py list (http://tinyurl.com/ybnb6p7 ), but I'm running into another problem. I'm trying to install the latest svn version with the stock sgimpi setup in mpi.cfg. I've also tried with intel's v10.1 compilers (v9.1 was the default). It compiles and installs just fine, but when I try to load mpi4py.MPI it can't find the MPI libraries. ---- he:jwise>python Python 2.6.3 (r263:75183, Oct 19 2009, 12:15:19) [GCC 3.3.3 (SuSE Linux)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import mpi4py.MPI Traceback (most recent call last): File "<stdin>", line 1, in <module> ImportError: /home/astro/jwise/local/lib/python2.6/site-packages/ mpi4py/MPI.so: undefined symbol: MPI_Comm_get_name ---- libmpi and libmpi++ are in /usr/lib, which is in my LD_LIBRARY_PATH (just in case... it should be included by default). If you haven't run into this problem before, maybe you (or someone else) could give me some tips on tracking down the mis-config. I haven't debugged something like this before... Thanks! John

2 2

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

yt-dev November 2009