Matt & Britton,
Commenting out __obtain_filenames will work as long as the paths in the
Commenting out __obtain_filenames in trunk doesn't seem to do it for me. My dataset is located where the full pathnames say they are. It's dying inside of _save_data in HierarchyType where it's trying to save the .yt file, due to h5py being unwilling to save an empty array (h5py.h5.ArgsError: Zero sized dimension for non-unlimited dimension). From pdb: -> arr = myGroup.create_dataset(name,data=array) (Pdb) name 'DataFields' (Pdb) array []
LAst night I discovered another set of data that was broken with the current trunk parser but works with the new hg parser -- there seems
This is a WoC-created dataset, so that may introduce more wrinkles into this.
...
I've spent the last while trying to get this to work in the hg yt tip, but I think I need some help. I've discovered a couple things in HierarchyType.py that seems wrong. I haven't committed them because I'm not sure I understand what's going on well enough to do that.
h5py doesn't like directly accessing the file pointer. Adding an intermediate step fixes this hangup:
diff -r d2386cb60c6d yt/lagos/HierarchyType.py
--- a/yt/lagos/HierarchyType.py Wed Nov 18 10:07:25 2009 -0800
+++ b/yt/lagos/HierarchyType.py Thu Nov 19 10:06:07 2009 -0800
@@ -345,7 +345,8 @@
"%s.hierarchy" % (pf.parameter_filename))
harray_fn = self.hierarchy_filename[:-9] + "harrays"
if os.path.exists(harray_fn):
- self.num_grids = h5py.File(harray_fn)["/Level"].len()
+ harray_fp = h5py.File(harray_fn)
+ self.num_grids = harray_fp["/Level"].len()
elif os.path.getsize(self.hierarchy_filename) == 0:
raise IOError(-1,"File empty", self.hierarchy_filename)
self.directory = os.path.dirname(self.hierarchy_filename)
These three datasets in the .harrays file are not 1D, they're 3D, so trying to put them in a flatiter array won't work. I might be doing something wrong here, but as it is doesn't seem quite right. Do we want to keep the self...arrays as flatiters?
@@ -483,9 +484,9 @@
def _parse_binary_hierarchy(self):
mylog.info("Getting the binary hierarchy")
f = h5py.File(self.hierarchy_filename[:-9] + "harrays")
- self.grid_dimensions.flat[:] = f["/ActiveDimensions"][:]
- self.grid_left_edge.flat[:] = f["/LeftEdges"][:]
- self.grid_right_edge.flat[:] = f["/RightEdges"][:]
+ self.grid_dimensions[:] = f["/ActiveDimensions"][:]
+ self.grid_left_edge[:] = f["/LeftEdges"][:]
+ self.grid_right_edge[:] = f["/RightEdges"][:]
levels = f["/Level"][:]
parents = f["/ParentIDs"][:]
procs = f["/Processor"][:]
With these fixes I'm hanging in the "for level in xrange(self.max_level+1):" loop in "_initialize_level_stats()" because self.max_level+1 is much much greater than the actual max:
(Pdb) self.max_level
2867
(Pdb) self.level_stats
rec.array([(8, 16777216, 0), (239, 256240, 1), (2, 944, 2), (1, 384, 3),
(3, 9536, 4), (2, 4992, 5), (2, 1488, 6), (1, 384, 7), (1, 384, 8),
(1, 216, 9), (1, 480, 10), (1, 384, 11), (2, 928, 12), (1, 512, 13),
(1, 288, 14), (3, 3200, 15), (1, 1200, 16), (1, 800, 17),
(1, 288, 18), (1, 512, 19), (1, 640, 20), (1, 384, 21),
(1, 216, 22), (1, 512, 23), (1, 288, 24), (1, 192, 25),
(1, 216, 26), (2, 8608, 27), (3, 4224, 28), (1, 384, 29),
(1, 1440, 30), (1, 640, 31), (1, 144, 32), (3, 1120, 33),
(1, 216, 34), (2, 728, 35), (1, 512, 36), (2, 432, 37),
(1, 384, 38), (1, 288, 39), (1, 288, 40), (1, 216, 41),
(2, 504, 42), (1, 5040, 43), (1, 3528, 44), (1, 1680, 45),
(1, 640, 46), (2, 1216, 47)],
dtype=[('numgrids', '