Mailman 3 May 2013 - yt-dev

Proposed changes to Octree code in yt 3.0
by Matthew Turk May 6, 2013

May 6, 2013

Hi all, I've shared some of this with ChrisM and SamL, but I wanted to bring it up here. This largely pertains to the codes RAMSES and NMSU-ART that are currently supported, as well as the support for particle Octrees. ARTIO is somewhat orthogonal at this time, as it doesn't store the octree in memory. Currently, Octs have: int (64-bits) local_ind (64-bits) domain (64-bits) pos[3] (64-bits each) level (64-bits) pointer-to-sd (64-bits) children, 8 pointers (64-bits each) parent (64-bits) … [View More]This comes out to 136 bytes. Much of this is because we have two different methods of traversing the Octs, both in allocation-order and in depth-first traversal order, and also because when I wrote this (seems like yesterday) I was trying to get something done without worrying too much about memory optimization. However, both of these things are becoming much more important; firstly, we need to ensure we're thinking about the memory implications of a large number of leaf nodes, and we need to verify that order of traversal is not important -- or at least, is guaranteed identical. Last week I spent some time rewriting the traversal code to attempt to get rid of much of the self-awareness of the octs, such that *outside* of a traversal, they are somewhat meaningless. This lets me cut it down to: domain index (which we may be able to eliminate in the future) code-specific index allocateable pointer to the 8 children (linearly) pointer-to-sd object This should be 32 bytes for leaf nodes, which expands for non-leaf nodes, although we may be able to cut that down as well (at the expense of slight over-allocation for instances when not all cells must be simultaneously refined.) I've implemented this with a traversal object, such that it will track the level state, the oct state, and so on and so forth. This works for all of the tests I have been able to create for it, but it results in a slight performance decrease. (I believe in the future we can optimize this.) Anyway, here's where I'm going with all of this. I'd like to extend the Octree base class, but in a way that leaves several items up to individual implementations. Furthermore, I'd like to start a brief discussion of how we might plan for this becoming distributed memory, which was impossible before but which I think will be possible by removing state from the Octs. = New Octree Base Class Methods = I'd like to propose these methods become part of the base octree class: * next_root(int domain_id, int ind[3]) * next_child(int domain_id) RAMSESOctreeContainer implements the first of these, and a version of the second of these items. (Which contains routines that are no longer necessary.) The distinction here is that the first of these routines will be able -- in the future, with no changes to API -- to add the root level oct to a sparse list of octs that belong to a given domain; this will, I hope, enable. Currently it will add them to a root_mesh attribute that lives on all the octrees. By using this, we'll be able to move more of the addition of new octs into the base Octree class, which should hide all of the logic for adding new octs from the individual implementations. [+-][01] on moving these explicitly to the base class and mandating their presence? Because we're thus going to keep a single method of traversing the octree, this will also mean that icoords, fcoords and ires can all be moved to the base class as well, which is the ultimate reason for the changes to the Octree anyway. = Sparse Octrees = it is my hope that we can stop allocating the "mask" variable that gets passed around and is (noct,8) in shape, and instead (if a mask is needed at all) only look at domain-by-domain information. For RAMSES and Gadget this will be a big improvement, but much less so for NMSU-ART and Tipsy, which are currently not set up for efficient multi-domain iteration. This will also mean that spatial chunking will be more efficient, and I think we'll be able to push mask selection deeper into the octree object itself. However, before we can start looking at distributed octrees with true sparse oct filling across nodes, we'll need to "pin" the parallelism. This would mean disabling the ability to add on new communicators, or possibly splitting workgroups up with an IO handler that communicates across MPI to the nodes that contain the sparse octree -- if we instantiate a sparse octree on N processors, we can't then move to subgroups for instance. So we'll need a way to pin it. But, I think with the changes here to removing global state and using a consistent mechanism for traversing the octree, we'll be able to do so eventually in the future. -Matt [View Less]

2 1

Units sprint
by Nathan Goldbaum May 6, 2013

May 6, 2013

Hi all, Casey Stark and I are planning on sprinting on the yt-3.0 units refactor next Thursday and Friday in Berkeley. If anyone is interested in joining (even remotely), let us know so we can keep you in the loop. -Nathan

2 1

[yt_analysis/yt] Rockstar Halos write_out (issue #558)
by Sam Skillman May 2, 2013

May 2, 2013

New issue 558: Rockstar Halos write_out https://bitbucket.org/yt_analysis/yt/issue/558/rockstar-halos-write_out Sam Skillman: Currently if you load halos using LoadRockstarHalos then try to write them out, it fails due to not knowing the redshift: ``` #!python In [14]: halos.write_out('halos.out') --------------------------------------------------------------------------- AttributeError Traceback (most recent call last) /projects/skillman/local/src/yt-hg/yt/mods.… [View More]

1 0

Fwd: [IPython-dev] A few notes on IPEPs
by Matthew Turk May 1, 2013

May 1, 2013

Since we've been discussing YTEPs lately, I thought I would forward this, which covers how IPython handles their similar items. We could discuss them in issues rather than pull requests, which would perhaps also be quite a nice fit for them. ---------- Forwarded message ---------- From: Brian Granger <ellisonbg(a)gmail.com> Date: Wed, May 1, 2013 at 7:12 PM Subject: [IPython-dev] A few notes on IPEPs To: IPython Development list <ipython-dev(a)scipy.org> Hi, Our usage of IPEPs … [View More]

1 0

Release of yt-2.5.2
by Matthew Turk May 1, 2013

May 1, 2013

Hi all, We're proud to release yt version 2.5.2. This is a scheduled point release that includes all bug fixes identified and fixed since the release of 2.5.1 on April 1. Changes and bug fixes include: * Upgrades to most of the dependencies in the install script * Speed improvements to ellipsoidal particle detection * Updates to FLASH fields * CASTRO frontend bug fixes * Fisheye camera bug fixes * ProjectionPlot now accepts a data_source keyword argument * Answer testing now … [View More]

1 0