In attempting to make a series of projections with halos plotted using the public install of yt on Kraken, I've encountered an odd problem. The job terminates without giving an apparent error statement in the output files, but rather ends with this cryptic (at least to me) mess: h5py._stub.IOError File "h5g.pyx", line 104, in h5py.h5g.GroupIter.__init__ (h5py/h5g.c:1248) File "h5g.pyx", line 104, in h5py.h5g.GroupIter.__init__ (h5py/h5g.c:1248) : Addr overflow (Invalid arguments to routine: Address overflowed) h5py._stub.IOError: Addr overflow (Invalid arguments to routine: Address overflowed) return func(*args, **kwds) File "h5g.pyx", line 317, in h5py.h5g.GroupID.get_num_objs (h5py/h5g.c:2482) return func(*args, **kwds) File "h5g.pyx", line 317, in h5py.h5g.GroupID.get_num_objs (h5py/h5g.c:2482) return func(*args, **kwds) File "h5g.pyx", line 317, in h5py.h5g.GroupID.get_num_objs (h5py/h5g.c:2482) return func(*args, **kwds) File "h5g.pyx", line 317, in h5py.h5g.GroupID.get_num_objs (h5py/h5g.c:2482) File "/lustre/scratch/proj/yt_common/trunk/lib/python2.6/site-packages/h5py-1.2.1-py2.6-linux-x86_64.egg/h5py/_sync.py", line 88, in wrap Does anyone have an idea what could be causing this problem? Thanks, Brian
Hi Brian, Can you tell us a little bit about the .yt file (or whichever file it's saving to)? Is it humongous, or does it have a ridiculous number of objects in it already? How many projections have you saved to it? How big is the sim? -Matt On Fri, Jul 23, 2010 at 11:04 AM, Brian Crosby <crosby.bd@gmail.com> wrote:
Matt, The sim is a 1024^3 box. They .yt file is pretty big, as it was created earlier during a halo finding and merger tree building job, but it's only creating 18 projections per dump. This error came about before even the first projection was created and saved, so there are none in there. Thanks, Brian On Jul 23, 2010, at 2:08 PM, Matthew Turk wrote:
Hi Brian, It might have too many groups -- I'm not sure. How big is pretty big, also? >2 gigs? >4 gigs? I'd suggest moving it out of the way for now and continuing on. Stephen, if you're listening, is it possible that the .yt file has had a ridiculous number of HDF5 groups made, which is confusing and overflowing h5py? -Matt On Fri, Jul 23, 2010 at 11:15 AM, Brian Crosby <crosby.bd@gmail.com> wrote:
Brian & Matt, Hello from Boulder! I'm probably not going to be able to follow this up very quickly. Until tomorrow my internet access is limited to cafés and we have lots of other things to get done!
Nothing that I've added to yt makes use of the .yt file, so the various tools I wrote Brian mentioned shouldn't affect it. One thought is I wonder if there are memory limits on the login nodes on Kraken, and if this is being hit? Brian, where you trying this on the login node, and if so, can you try it in an interactive session (one node will suffice). _______________________________________________________ sskory@physics.ucsd.edu o__ Stephen Skory http://physics.ucsd.edu/~sskory/ _.>/ _Graduate Student ________________________________(_)_\(_)_______________
Stephen, This was being run as a batch job, 2 cores/node on 256 cores total. I just tried loading up yt on the login node of Kraken, and hit some problems. Does any of this illuminate things? Python 2.4.2 (#1, Dec 2 2008, 00:09:07) [GCC 4.1.2 20070115 (SUSE Linux)] on linux2 Type "help", "copyright", "credits" or "license" for more information.
-Brian On Jul 23, 2010, at 4:23 PM, Stephen Skory wrote:
I've only been sparsely following this thread, but I noticed that you're running Python 2.4.2 on Kraken, whereas you're using yt in a directory labeled python2.6. Could a mismatch of python versions cause such a problem? Isn't the default version of python on kraken 2.6.4? You could type "which python" to find out which version of python you're using, but 2.6.4 is in: /sw/xt/python/2.6.4/sles10.1_gnu4.1.2//bin/python If you want to switch to using 2.6.4, you could type: $ module load python/2.6.4 Matt or Esteban, could a mismatch of python versions cause this problem in yt? Or am I totally off-base here? Cameron Brian Crosby wrote:
-- Cameron Hummels PhD Candidate, Astronomy Department of Columbia University Public Outreach Director, Astronomy Department of Columbia University NASA IYA New York State Student Ambassador http://outreach.astro.columbia.edu
Hi Brian, Can you tell us a little bit about the .yt file (or whichever file it's saving to)? Is it humongous, or does it have a ridiculous number of objects in it already? How many projections have you saved to it? How big is the sim? -Matt On Fri, Jul 23, 2010 at 11:04 AM, Brian Crosby <crosby.bd@gmail.com> wrote:
Matt, The sim is a 1024^3 box. They .yt file is pretty big, as it was created earlier during a halo finding and merger tree building job, but it's only creating 18 projections per dump. This error came about before even the first projection was created and saved, so there are none in there. Thanks, Brian On Jul 23, 2010, at 2:08 PM, Matthew Turk wrote:
Hi Brian, It might have too many groups -- I'm not sure. How big is pretty big, also? >2 gigs? >4 gigs? I'd suggest moving it out of the way for now and continuing on. Stephen, if you're listening, is it possible that the .yt file has had a ridiculous number of HDF5 groups made, which is confusing and overflowing h5py? -Matt On Fri, Jul 23, 2010 at 11:15 AM, Brian Crosby <crosby.bd@gmail.com> wrote:
Brian & Matt, Hello from Boulder! I'm probably not going to be able to follow this up very quickly. Until tomorrow my internet access is limited to cafés and we have lots of other things to get done!
Nothing that I've added to yt makes use of the .yt file, so the various tools I wrote Brian mentioned shouldn't affect it. One thought is I wonder if there are memory limits on the login nodes on Kraken, and if this is being hit? Brian, where you trying this on the login node, and if so, can you try it in an interactive session (one node will suffice). _______________________________________________________ sskory@physics.ucsd.edu o__ Stephen Skory http://physics.ucsd.edu/~sskory/ _.>/ _Graduate Student ________________________________(_)_\(_)_______________
Stephen, This was being run as a batch job, 2 cores/node on 256 cores total. I just tried loading up yt on the login node of Kraken, and hit some problems. Does any of this illuminate things? Python 2.4.2 (#1, Dec 2 2008, 00:09:07) [GCC 4.1.2 20070115 (SUSE Linux)] on linux2 Type "help", "copyright", "credits" or "license" for more information.
-Brian On Jul 23, 2010, at 4:23 PM, Stephen Skory wrote:
I've only been sparsely following this thread, but I noticed that you're running Python 2.4.2 on Kraken, whereas you're using yt in a directory labeled python2.6. Could a mismatch of python versions cause such a problem? Isn't the default version of python on kraken 2.6.4? You could type "which python" to find out which version of python you're using, but 2.6.4 is in: /sw/xt/python/2.6.4/sles10.1_gnu4.1.2//bin/python If you want to switch to using 2.6.4, you could type: $ module load python/2.6.4 Matt or Esteban, could a mismatch of python versions cause this problem in yt? Or am I totally off-base here? Cameron Brian Crosby wrote:
-- Cameron Hummels PhD Candidate, Astronomy Department of Columbia University Public Outreach Director, Astronomy Department of Columbia University NASA IYA New York State Student Ambassador http://outreach.astro.columbia.edu
participants (4)
-
Brian Crosby -
Cameron Hummels -
Matthew Turk -
Stephen Skory