I have another odd issue, again stemming from with this vintage dataset. It's a 512^3 root with 4 levels, and a refinement by 4.
When the parallel analysis gave me problems yesterday, I tried re-griding the data by restarting and forcing parallel root grid io on (including a removal of the rebuild hierarchy call to ensure the grids stay the same, and returning before any hydro is done.) The odd bit is now projections look different. Attached are two images, sphere_serial is through the old serial root grid run, sphere_parallel is through the output with parallel root grid io. Centered on the max, 0.02 in radius, Density projection. Projections were made using proj = pf.h.proj(ax,field,center=center,source=sphere) You can see that less structure seems to be captured in the parallel dataset.
I've done two checks to make sure that it's not a glitch in the re-griding. First is the actual data-- the grids are in a different order from parallel to serial, but once that's taken into account all the data in subgrids is identical (at least in density, the field in question). Additionally the new tiles match the corresponding positions in the old tile. So the data itself is fine.
The other test I did was to regrid, but leaving serial root grid io on-- new serial set is identical (except again for grid order) to the old, and the projections match.
Differences also show up in profiles.
Again, not a major problem, but it's pretty counter-intuitive, so I was wondering if anyone had run into this.