Hi Nathan,
On Tue, Jan 17, 2012 at 8:49 PM, Nathan Goldbaum
Hi all,
I'm having a curious issue. I'm trying to open a large number of FLASH parameter files, read in some vital statistics about each file, and then write out the data to a ascii table.
However, for some reason I'm finding that this crashes after 1020 data files have been opened. There's nothing special about the 1020th file since I can open it and inspect it interactively. This may have something to do with me running this script on the lustre filesystem on Pleiades at NASA Ames.
This is puzzling. My guess is that something about HDF5's internal reference counting, or h5py, or something like that is causing these issues. My initial guess for a workaround would be to try os.fork, but having now tested it I'm not sure that will reset the file handles and so forth.
My test script is pasted here:
http://paste.yt-project.org/show/2039/
And a sample traceback is pasted here:
http://paste.yt-project.org/show/2040/
It looks like the crash happens when yt tries to open the h5py handle for the data file. It crashes as if the data file doesn't exist, even though it really does, I promise. This seems to be related to an h5py issue that John ZuHone noticed back in July:
http://lists.spacepope.org/htdig.cgi/yt-users-spacepope.org/2011-July/001703...
Adding 'del pf._handle' to the end of my loop does not fix the issue.
Can you reproduce it inside a small loop, without yt, just using h5py and calling .close() on each file? -Matt
Thanks for your help with this!
Nathan Goldbaum Graduate Student Astronomy & Astrophysics, UCSC goldbaum@ucolick.org http://www.ucolick.org/~goldbaum
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org