Hi all, I've just encountered the same h5py bug in another place. This is with h5py 1.2.0. The newest is 1.3.1. What do we think about upgrading? -- Stephen Skory s@skory.us http://stephenskory.com/ 510.621.3687 (google voice)
Hi Stephen,
I am generally opposed to upgrading either Matplotlib or h5py without
a very careful and detailed study of how the API changes. The
transition between 1.1 and 1.2 of h5py included a number of API
breakages that affected us, and we saw the same thing with 0.9x to
0.9[89] and now 1.0 to 1.0.1 in matplotlib.
Have you identified the smallest possible set of code that replicates
this, as you called it, apparent bug? Did you test it against 1.3.1?
Please also consider reporting it upstream.
-Matt
On Mon, Apr 11, 2011 at 8:08 PM, Stephen Skory wrote:
Hi all,
I've just encountered the same h5py bug in another place. This is with h5py 1.2.0. The newest is 1.3.1. What do we think about upgrading?
-- Stephen Skory s@skory.us http://stephenskory.com/ 510.621.3687 (google voice) _______________________________________________ Yt-dev mailing list Yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
Matt,
Have you identified the smallest possible set of code that replicates this, as you called it, apparent bug? Did you test it against 1.3.1? Please also consider reporting it upstream.
I'll report it upstream if I can replicate it in a simple script. I'll also give 1.2.1 a shot, which is also available. I'll let you know. -- Stephen Skory s@skory.us http://stephenskory.com/ 510.621.3687 (google voice)
Hi all,
I'll report it upstream if I can replicate it in a simple script. I'll also give 1.2.1 a shot, which is also available. I'll let you know.
After some testing, 1.2.1 does not fix the error. I've also found that a simple script like this will make the error: import h5py f = h5py.File('data0090.cpu0000') f.close() f = h5py.File('data0090.cpu0000') print f['Grid00000001']['particle_position_x'][:] f.close() Can someone try this on some data to see if they get the error too? Thanks! -- Stephen Skory s@skory.us http://stephenskory.com/ 510.621.3687 (google voice)
Hi Stephen, I got the error on the print f['Grid00000001']['particle_position_x'][:] line, same as before, it didn't even get to the 2nd f.close() So I take it the error is in h5py but your patch in YT only fixes it in the case of loading halo data? From G.S.
Hi all,
I'll report it upstream if I can replicate it in a simple script. I'll also give 1.2.1 a shot, which is also available. I'll let you know.
After some testing, 1.2.1 does not fix the error. I've also found that a simple script like this will make the error:
import h5py f = h5py.File('data0090.cpu0000') f.close() f = h5py.File('data0090.cpu0000') print f['Grid00000001']['particle_position_x'][:] f.close()
Can someone try this on some data to see if they get the error too? Thanks!
-- Stephen Skory s@skory.us http://stephenskory.com/ 510.621.3687 (google voice) _______________________________________________ Yt-dev mailing list Yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
Hi Stephen,
Yes, this errors out for me.
What it looks like is that the deallocation routine is being deferred.
HDF5 keeps a global state of files, and the order in which the
variable is dereferences, the file objects are closed, and the new
file is assigned is conflicting. In short, what I *think* is going
on, but which I may be wrong about:
h5py.File("whatever") is being evaluated.
Subsequently, it is assigned to variable 'f'
Then, as a result of this, the *old* variable 'f''s reference count
drops to zero. This calls the deallocation routine.
However, at this point, the file has been re-opened by a different variable.
This is all fixed if you use a different variable name for the second opening:
import h5py
f = h5py.File('test.h5')
f.close()
f2 = h5py.File('test.h5')
print f2['Grid00000001']['particle_position_x'][:]
f2.close()
If the variable goes out of scope, the same thing should work.
Additionally, the second 'del f' on line 881 of halo_objects should
not be necessary for this fix. Can you see if it is?
Perhaps more descriptive variable names (something I too am guilty of,
all too often) would obviate the need for all the del's.
-Matt
On Mon, Apr 11, 2011 at 10:22 PM, Stephen Skory wrote:
Hi all,
I'll report it upstream if I can replicate it in a simple script. I'll also give 1.2.1 a shot, which is also available. I'll let you know.
After some testing, 1.2.1 does not fix the error. I've also found that a simple script like this will make the error:
import h5py f = h5py.File('data0090.cpu0000') f.close() f = h5py.File('data0090.cpu0000') print f['Grid00000001']['particle_position_x'][:] f.close()
Can someone try this on some data to see if they get the error too? Thanks!
-- Stephen Skory s@skory.us http://stephenskory.com/ 510.621.3687 (google voice) _______________________________________________ Yt-dev mailing list Yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
Hi Stephen and everyone else,
Stephen's right, this is a bug; I rescind my earlier conservatism and
hereby eat crow. He did the work of looking for API changes, of which
it seems there are none, so I have updated the install script to h5py
1.3.1. I still think we need to account for people using 1.2.0, which
means being more careful about our variable names.
-Matt
On Mon, Apr 11, 2011 at 10:32 PM, Matthew Turk
Hi Stephen,
Yes, this errors out for me.
What it looks like is that the deallocation routine is being deferred. HDF5 keeps a global state of files, and the order in which the variable is dereferences, the file objects are closed, and the new file is assigned is conflicting. In short, what I *think* is going on, but which I may be wrong about:
h5py.File("whatever") is being evaluated. Subsequently, it is assigned to variable 'f' Then, as a result of this, the *old* variable 'f''s reference count drops to zero. This calls the deallocation routine. However, at this point, the file has been re-opened by a different variable.
This is all fixed if you use a different variable name for the second opening:
import h5py f = h5py.File('test.h5') f.close() f2 = h5py.File('test.h5') print f2['Grid00000001']['particle_position_x'][:] f2.close()
If the variable goes out of scope, the same thing should work. Additionally, the second 'del f' on line 881 of halo_objects should not be necessary for this fix. Can you see if it is?
Perhaps more descriptive variable names (something I too am guilty of, all too often) would obviate the need for all the del's.
-Matt
On Mon, Apr 11, 2011 at 10:22 PM, Stephen Skory
wrote:Hi all,
I'll report it upstream if I can replicate it in a simple script. I'll also give 1.2.1 a shot, which is also available. I'll let you know.
After some testing, 1.2.1 does not fix the error. I've also found that a simple script like this will make the error:
import h5py f = h5py.File('data0090.cpu0000') f.close() f = h5py.File('data0090.cpu0000') print f['Grid00000001']['particle_position_x'][:] f.close()
Can someone try this on some data to see if they get the error too? Thanks!
-- Stephen Skory s@skory.us http://stephenskory.com/ 510.621.3687 (google voice) _______________________________________________ Yt-dev mailing list Yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
I've just encountered the same h5py bug in another place. This is with h5py 1.2.0. The newest is 1.3.1. What do we think about upgrading?
As a follow-up, I just tried things with 1.3.1, and I didn't get the crash. -- Stephen Skory s@skory.us http://stephenskory.com/ 510.621.3687 (google voice)
participants (3)
-
gso@physics.ucsd.edu
-
Matthew Turk
-
Stephen Skory