error while trying to find abundances through checkpoint files
HI people, I am using yt 3.3.1 and submitting my SLURM script to stampede. I was using this script to find abundances of C, O etc through checkpoint files in FLASH. while my script worked fine with old yt (3.1) , suddenly it crashed today and returned me the following error: *yt : [INFO ] 2016-10-17 23:22:24,295 Parameters: current_time = 28.1847530806* *yt : [INFO ] 2016-10-17 23:22:24,295 Parameters: domain_dimensions = [128 128 128]* *yt : [INFO ] 2016-10-17 23:22:24,296 Parameters: domain_left_edge = [ -2.80000000e+10 -2.80000000e+10 -2.80000000e+10]* *yt : [INFO ] 2016-10-17 23:22:24,296 Parameters: domain_right_edge = [ 2.80000000e+10 2.80000000e+10 2.80000000e+10]* *yt : [INFO ] 2016-10-17 23:22:24,296 Parameters: cosmological_simulation = 0.0* *Executin lessg abundance.py* *Traceback (most recent call last):* * File "abundance2.py", line 304, in <module>* * main(chkFilenames_own)* * File "abundance2.py", line 59, in main* * pf = yt.load(filenames[n])* * File "/work/03858/thaque56/sw/yt-new-3.3/yt-conda/lib/python2.7/site-packages/yt/convenience.py", line 79, in load* * if c._is_valid(*args, **kwargs): candidates.append(n)* * File "/work/03858/thaque56/sw/yt-new-3.3/yt-conda/lib/python2.7/site-packages/yt/frontends/flash/data_structures.py", line 478, in _is_valid* * if "bounding box" not in fileh["/"].keys() \* * File "/work/03858/thaque56/sw/yt-new-3.3/yt-conda/lib/python2.7/site-packages/h5py/_hl/base.py", line 368, in keys* * return list(self)* * File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper (/home/ilan/minonda/conda-bld/work/h5py/_objects.c:2696)* * File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper (/home/ilan/minonda/conda-bld/work/h5py/_objects.c:2654)* * File "/work/03858/thaque56/sw/yt-new-3.3/yt-conda/lib/python2.7/site-packages/h5py/_hl/group.py", line 303, in __len__* * return self.id.get_num_objs()* * File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper (/home/ilan/minonda/conda-bld/work/h5py/_objects.c:2696)* * File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper (/home/ilan/minonda/conda-bld/work/h5py/_objects.c:2654)* * File "h5py/h5g.pyx", line 321, in h5py.h5g.GroupID.get_num_objs (/home/ilan/minonda/conda-bld/work/h5py/h5g.c:4194)* *RuntimeError: Can't determine (Bad symbol table node signature)* *[c560-102.stampede.tacc.utexas.edu:mpispawn_0][child_handler] MPI process (rank: 0, pid: 22137) exited with status 1* *TACC: MPI job exited with code: 1* *TACC: Shutdown complete. Exiting.* I was wondering if there is something wrong with my code or the new yt. I am also attaching my code here to look at. Thanks in advance Best Tazkera
Is the script being run in parallel? If so, does it crash if you run it on only one core? Nathan On Monday, October 17, 2016, tazkera haque <h.tazkera@gmail.com> wrote:
HI people,
I am using yt 3.3.1 and submitting my SLURM script to stampede. I was using this script to find abundances of C, O etc through checkpoint files in FLASH. while my script worked fine with old yt (3.1) , suddenly it crashed today and returned me the following error:
*yt : [INFO ] 2016-10-17 23:22:24,295 Parameters: current_time = 28.1847530806* *yt : [INFO ] 2016-10-17 23:22:24,295 Parameters: domain_dimensions = [128 128 128]* *yt : [INFO ] 2016-10-17 23:22:24,296 Parameters: domain_left_edge = [ -2.80000000e+10 -2.80000000e+10 -2.80000000e+10]* *yt : [INFO ] 2016-10-17 23:22:24,296 Parameters: domain_right_edge = [ 2.80000000e+10 2.80000000e+10 2.80000000e+10]* *yt : [INFO ] 2016-10-17 23:22:24,296 Parameters: cosmological_simulation = 0.0* *Executin lessg abundance.py* *Traceback (most recent call last):* * File "abundance2.py", line 304, in <module>* * main(chkFilenames_own)* * File "abundance2.py", line 59, in main* * pf = yt.load(filenames[n])* * File "/work/03858/thaque56/sw/yt-new-3.3/yt-conda/lib/python2.7/site-packages/yt/convenience.py", line 79, in load* * if c._is_valid(*args, **kwargs): candidates.append(n)* * File "/work/03858/thaque56/sw/yt-new-3.3/yt-conda/lib/python2.7/site-packages/yt/frontends/flash/data_structures.py", line 478, in _is_valid* * if "bounding box" not in fileh["/"].keys() \* * File "/work/03858/thaque56/sw/yt-new-3.3/yt-conda/lib/python2.7/site-packages/h5py/_hl/base.py", line 368, in keys* * return list(self)* * File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper (/home/ilan/minonda/conda-bld/work/h5py/_objects.c:2696)* * File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper (/home/ilan/minonda/conda-bld/work/h5py/_objects.c:2654)* * File "/work/03858/thaque56/sw/yt-new-3.3/yt-conda/lib/python2.7/site-packages/h5py/_hl/group.py", line 303, in __len__* * return self.id.get_num_objs()* * File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper (/home/ilan/minonda/conda-bld/work/h5py/_objects.c:2696)* * File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper (/home/ilan/minonda/conda-bld/work/h5py/_objects.c:2654)* * File "h5py/h5g.pyx", line 321, in h5py.h5g.GroupID.get_num_objs (/home/ilan/minonda/conda-bld/work/h5py/h5g.c:4194)* *RuntimeError: Can't determine (Bad symbol table node signature)* *[c560-102.stampede.tacc.utexas.edu:mpispawn_0][child_handler] MPI process (rank: 0, pid: 22137) exited with status 1* *TACC: MPI job exited with code: 1*
*TACC: Shutdown complete. Exiting.*
I was wondering if there is something wrong with my code or the new yt. I am also attaching my code here to look at. Thanks in advance
Best Tazkera
yes it's being run in parallel, I didn't check with one core yet, I will let you know what happens then On Tue, Oct 18, 2016 at 12:50 AM, Nathan Goldbaum <nathan12343@gmail.com> wrote:
Is the script being run in parallel? If so, does it crash if you run it on only one core?
Nathan
On Monday, October 17, 2016, tazkera haque <h.tazkera@gmail.com> wrote:
HI people,
I am using yt 3.3.1 and submitting my SLURM script to stampede. I was using this script to find abundances of C, O etc through checkpoint files in FLASH. while my script worked fine with old yt (3.1) , suddenly it crashed today and returned me the following error:
*yt : [INFO ] 2016-10-17 23:22:24,295 Parameters: current_time = 28.1847530806* *yt : [INFO ] 2016-10-17 23:22:24,295 Parameters: domain_dimensions = [128 128 128]* *yt : [INFO ] 2016-10-17 23:22:24,296 Parameters: domain_left_edge = [ -2.80000000e+10 -2.80000000e+10 -2.80000000e+10]* *yt : [INFO ] 2016-10-17 23:22:24,296 Parameters: domain_right_edge = [ 2.80000000e+10 2.80000000e+10 2.80000000e+10]* *yt : [INFO ] 2016-10-17 23:22:24,296 Parameters: cosmological_simulation = 0.0* *Executin lessg abundance.py* *Traceback (most recent call last):* * File "abundance2.py", line 304, in <module>* * main(chkFilenames_own)* * File "abundance2.py", line 59, in main* * pf = yt.load(filenames[n])* * File "/work/03858/thaque56/sw/yt-new-3.3/yt-conda/lib/python2.7/site-packages/yt/convenience.py", line 79, in load* * if c._is_valid(*args, **kwargs): candidates.append(n)* * File "/work/03858/thaque56/sw/yt-new-3.3/yt-conda/lib/python2.7/site-packages/yt/frontends/flash/data_structures.py", line 478, in _is_valid* * if "bounding box" not in fileh["/"].keys() \* * File "/work/03858/thaque56/sw/yt-new-3.3/yt-conda/lib/python2.7/site-packages/h5py/_hl/base.py", line 368, in keys* * return list(self)* * File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper (/home/ilan/minonda/conda-bld/work/h5py/_objects.c:2696)* * File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper (/home/ilan/minonda/conda-bld/work/h5py/_objects.c:2654)* * File "/work/03858/thaque56/sw/yt-new-3.3/yt-conda/lib/python2.7/site-packages/h5py/_hl/group.py", line 303, in __len__* * return self.id.get_num_objs()* * File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper (/home/ilan/minonda/conda-bld/work/h5py/_objects.c:2696)* * File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper (/home/ilan/minonda/conda-bld/work/h5py/_objects.c:2654)* * File "h5py/h5g.pyx", line 321, in h5py.h5g.GroupID.get_num_objs (/home/ilan/minonda/conda-bld/work/h5py/h5g.c:4194)* *RuntimeError: Can't determine (Bad symbol table node signature)* *[c560-102.stampede.tacc.utexas.edu:mpispawn_0][child_handler] MPI process (rank: 0, pid: 22137) exited with status 1* *TACC: MPI job exited with code: 1*
*TACC: Shutdown complete. Exiting.*
I was wondering if there is something wrong with my code or the new yt. I am also attaching my code here to look at. Thanks in advance
Best Tazkera
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
Hi Nathan, I tried with one file in my ipython notebook, it seems to work there On Tue, Oct 18, 2016 at 12:54 AM, tazkera haque <h.tazkera@gmail.com> wrote:
yes it's being run in parallel, I didn't check with one core yet, I will let you know what happens then
On Tue, Oct 18, 2016 at 12:50 AM, Nathan Goldbaum <nathan12343@gmail.com> wrote:
Is the script being run in parallel? If so, does it crash if you run it on only one core?
Nathan
On Monday, October 17, 2016, tazkera haque <h.tazkera@gmail.com> wrote:
HI people,
I am using yt 3.3.1 and submitting my SLURM script to stampede. I was using this script to find abundances of C, O etc through checkpoint files in FLASH. while my script worked fine with old yt (3.1) , suddenly it crashed today and returned me the following error:
*yt : [INFO ] 2016-10-17 23:22:24,295 Parameters: current_time = 28.1847530806* *yt : [INFO ] 2016-10-17 23:22:24,295 Parameters: domain_dimensions = [128 128 128]* *yt : [INFO ] 2016-10-17 23:22:24,296 Parameters: domain_left_edge = [ -2.80000000e+10 -2.80000000e+10 -2.80000000e+10]* *yt : [INFO ] 2016-10-17 23:22:24,296 Parameters: domain_right_edge = [ 2.80000000e+10 2.80000000e+10 2.80000000e+10]* *yt : [INFO ] 2016-10-17 23:22:24,296 Parameters: cosmological_simulation = 0.0* *Executin lessg abundance.py* *Traceback (most recent call last):* * File "abundance2.py", line 304, in <module>* * main(chkFilenames_own)* * File "abundance2.py", line 59, in main* * pf = yt.load(filenames[n])* * File "/work/03858/thaque56/sw/yt-new-3.3/yt-conda/lib/python2.7/site-packages/yt/convenience.py", line 79, in load* * if c._is_valid(*args, **kwargs): candidates.append(n)* * File "/work/03858/thaque56/sw/yt-new-3.3/yt-conda/lib/python2.7/site-packages/yt/frontends/flash/data_structures.py", line 478, in _is_valid* * if "bounding box" not in fileh["/"].keys() \* * File "/work/03858/thaque56/sw/yt-new-3.3/yt-conda/lib/python2.7/site-packages/h5py/_hl/base.py", line 368, in keys* * return list(self)* * File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper (/home/ilan/minonda/conda-bld/work/h5py/_objects.c:2696)* * File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper (/home/ilan/minonda/conda-bld/work/h5py/_objects.c:2654)* * File "/work/03858/thaque56/sw/yt-new-3.3/yt-conda/lib/python2.7/site-packages/h5py/_hl/group.py", line 303, in __len__* * return self.id.get_num_objs()* * File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper (/home/ilan/minonda/conda-bld/work/h5py/_objects.c:2696)* * File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper (/home/ilan/minonda/conda-bld/work/h5py/_objects.c:2654)* * File "h5py/h5g.pyx", line 321, in h5py.h5g.GroupID.get_num_objs (/home/ilan/minonda/conda-bld/work/h5py/h5g.c:4194)* *RuntimeError: Can't determine (Bad symbol table node signature)* *[c560-102.stampede.tacc.utexas.edu:mpispawn_0][child_handler] MPI process (rank: 0, pid: 22137) exited with status 1* *TACC: MPI job exited with code: 1*
*TACC: Shutdown complete. Exiting.*
I was wondering if there is something wrong with my code or the new yt. I am also attaching my code here to look at. Thanks in advance
Best Tazkera
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
HI Nathan, I figured out what was going wrong, I submitted my job script from the $SCRATCH folder. apparently jobs can only be submitted through the $WORK folder on Stampede. thanks for your prompt response though On Tue, Oct 18, 2016 at 1:22 AM, tazkera haque <h.tazkera@gmail.com> wrote:
Hi Nathan,
I tried with one file in my ipython notebook, it seems to work there
On Tue, Oct 18, 2016 at 12:54 AM, tazkera haque <h.tazkera@gmail.com> wrote:
yes it's being run in parallel, I didn't check with one core yet, I will let you know what happens then
On Tue, Oct 18, 2016 at 12:50 AM, Nathan Goldbaum <nathan12343@gmail.com> wrote:
Is the script being run in parallel? If so, does it crash if you run it on only one core?
Nathan
On Monday, October 17, 2016, tazkera haque <h.tazkera@gmail.com> wrote:
HI people,
I am using yt 3.3.1 and submitting my SLURM script to stampede. I was using this script to find abundances of C, O etc through checkpoint files in FLASH. while my script worked fine with old yt (3.1) , suddenly it crashed today and returned me the following error:
*yt : [INFO ] 2016-10-17 23:22:24,295 Parameters: current_time = 28.1847530806* *yt : [INFO ] 2016-10-17 23:22:24,295 Parameters: domain_dimensions = [128 128 128]* *yt : [INFO ] 2016-10-17 23:22:24,296 Parameters: domain_left_edge = [ -2.80000000e+10 -2.80000000e+10 -2.80000000e+10]* *yt : [INFO ] 2016-10-17 23:22:24,296 Parameters: domain_right_edge = [ 2.80000000e+10 2.80000000e+10 2.80000000e+10]* *yt : [INFO ] 2016-10-17 23:22:24,296 Parameters: cosmological_simulation = 0.0* *Executin lessg abundance.py* *Traceback (most recent call last):* * File "abundance2.py", line 304, in <module>* * main(chkFilenames_own)* * File "abundance2.py", line 59, in main* * pf = yt.load(filenames[n])* * File "/work/03858/thaque56/sw/yt-new-3.3/yt-conda/lib/python2.7/site-packages/yt/convenience.py", line 79, in load* * if c._is_valid(*args, **kwargs): candidates.append(n)* * File "/work/03858/thaque56/sw/yt-new-3.3/yt-conda/lib/python2.7/site-packages/yt/frontends/flash/data_structures.py", line 478, in _is_valid* * if "bounding box" not in fileh["/"].keys() \* * File "/work/03858/thaque56/sw/yt-new-3.3/yt-conda/lib/python2.7/site-packages/h5py/_hl/base.py", line 368, in keys* * return list(self)* * File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper (/home/ilan/minonda/conda-bld/work/h5py/_objects.c:2696)* * File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper (/home/ilan/minonda/conda-bld/work/h5py/_objects.c:2654)* * File "/work/03858/thaque56/sw/yt-new-3.3/yt-conda/lib/python2.7/site-packages/h5py/_hl/group.py", line 303, in __len__* * return self.id.get_num_objs()* * File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper (/home/ilan/minonda/conda-bld/work/h5py/_objects.c:2696)* * File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper (/home/ilan/minonda/conda-bld/work/h5py/_objects.c:2654)* * File "h5py/h5g.pyx", line 321, in h5py.h5g.GroupID.get_num_objs (/home/ilan/minonda/conda-bld/work/h5py/h5g.c:4194)* *RuntimeError: Can't determine (Bad symbol table node signature)* *[c560-102.stampede.tacc.utexas.edu:mpispawn_0][child_handler] MPI process (rank: 0, pid: 22137) exited with status 1* *TACC: MPI job exited with code: 1*
*TACC: Shutdown complete. Exiting.*
I was wondering if there is something wrong with my code or the new yt. I am also attaching my code here to look at. Thanks in advance
Best Tazkera
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
HI Nathan, Sorry to bother you again, but the problem seems to prevail even working from $WORK directory. I got the same error msg this morning again with a different script. Do you see anything wrong with the script I attached ? I have used it for a long time now without any sort of error. Thanks On Tue, Oct 18, 2016 at 2:34 AM, tazkera haque <h.tazkera@gmail.com> wrote:
HI Nathan,
I figured out what was going wrong, I submitted my job script from the $SCRATCH folder. apparently jobs can only be submitted through the $WORK folder on Stampede. thanks for your prompt response though
On Tue, Oct 18, 2016 at 1:22 AM, tazkera haque <h.tazkera@gmail.com> wrote:
Hi Nathan,
I tried with one file in my ipython notebook, it seems to work there
On Tue, Oct 18, 2016 at 12:54 AM, tazkera haque <h.tazkera@gmail.com> wrote:
yes it's being run in parallel, I didn't check with one core yet, I will let you know what happens then
On Tue, Oct 18, 2016 at 12:50 AM, Nathan Goldbaum <nathan12343@gmail.com
wrote:
Is the script being run in parallel? If so, does it crash if you run it on only one core?
Nathan
On Monday, October 17, 2016, tazkera haque <h.tazkera@gmail.com> wrote:
HI people,
I am using yt 3.3.1 and submitting my SLURM script to stampede. I was using this script to find abundances of C, O etc through checkpoint files in FLASH. while my script worked fine with old yt (3.1) , suddenly it crashed today and returned me the following error:
*yt : [INFO ] 2016-10-17 23:22:24,295 Parameters: current_time = 28.1847530806* *yt : [INFO ] 2016-10-17 23:22:24,295 Parameters: domain_dimensions = [128 128 128]* *yt : [INFO ] 2016-10-17 23:22:24,296 Parameters: domain_left_edge = [ -2.80000000e+10 -2.80000000e+10 -2.80000000e+10]* *yt : [INFO ] 2016-10-17 23:22:24,296 Parameters: domain_right_edge = [ 2.80000000e+10 2.80000000e+10 2.80000000e+10]* *yt : [INFO ] 2016-10-17 23:22:24,296 Parameters: cosmological_simulation = 0.0* *Executin lessg abundance.py* *Traceback (most recent call last):* * File "abundance2.py", line 304, in <module>* * main(chkFilenames_own)* * File "abundance2.py", line 59, in main* * pf = yt.load(filenames[n])* * File "/work/03858/thaque56/sw/yt-new-3.3/yt-conda/lib/python2.7/site-packages/yt/convenience.py", line 79, in load* * if c._is_valid(*args, **kwargs): candidates.append(n)* * File "/work/03858/thaque56/sw/yt-new-3.3/yt-conda/lib/python2.7/site-packages/yt/frontends/flash/data_structures.py", line 478, in _is_valid* * if "bounding box" not in fileh["/"].keys() \* * File "/work/03858/thaque56/sw/yt-new-3.3/yt-conda/lib/python2.7/site-packages/h5py/_hl/base.py", line 368, in keys* * return list(self)* * File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper (/home/ilan/minonda/conda-bld/work/h5py/_objects.c:2696)* * File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper (/home/ilan/minonda/conda-bld/work/h5py/_objects.c:2654)* * File "/work/03858/thaque56/sw/yt-new-3.3/yt-conda/lib/python2.7/site-packages/h5py/_hl/group.py", line 303, in __len__* * return self.id.get_num_objs()* * File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper (/home/ilan/minonda/conda-bld/work/h5py/_objects.c:2696)* * File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper (/home/ilan/minonda/conda-bld/work/h5py/_objects.c:2654)* * File "h5py/h5g.pyx", line 321, in h5py.h5g.GroupID.get_num_objs (/home/ilan/minonda/conda-bld/work/h5py/h5g.c:4194)* *RuntimeError: Can't determine (Bad symbol table node signature)* *[c560-102.stampede.tacc.utexas.edu:mpispawn_0][child_handler] MPI process (rank: 0, pid: 22137) exited with status 1* *TACC: MPI job exited with code: 1*
*TACC: Shutdown complete. Exiting.*
I was wondering if there is something wrong with my code or the new yt. I am also attaching my code here to look at. Thanks in advance
Best Tazkera
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
Hi Tazkera, When I tried googling your error message last night, I found that it's associated with more than one MPI process trying to access an HDF5 file at the same time, presumably using a serial version of the HDF5 library. I just did a quick test and I'm unable to reproduce here on my laptop. Unfortunately I don't have access to stampede so can't reproduce there. Can you share exactly which h5py and HDF5 library versions you're using? -Nathan On Tue, Oct 18, 2016 at 3:05 PM, tazkera haque <h.tazkera@gmail.com> wrote:
HI Nathan,
Sorry to bother you again, but the problem seems to prevail even working from $WORK directory. I got the same error msg this morning again with a different script. Do you see anything wrong with the script I attached ? I have used it for a long time now without any sort of error.
Thanks
On Tue, Oct 18, 2016 at 2:34 AM, tazkera haque <h.tazkera@gmail.com> wrote:
HI Nathan,
I figured out what was going wrong, I submitted my job script from the $SCRATCH folder. apparently jobs can only be submitted through the $WORK folder on Stampede. thanks for your prompt response though
On Tue, Oct 18, 2016 at 1:22 AM, tazkera haque <h.tazkera@gmail.com> wrote:
Hi Nathan,
I tried with one file in my ipython notebook, it seems to work there
On Tue, Oct 18, 2016 at 12:54 AM, tazkera haque <h.tazkera@gmail.com> wrote:
yes it's being run in parallel, I didn't check with one core yet, I will let you know what happens then
On Tue, Oct 18, 2016 at 12:50 AM, Nathan Goldbaum < nathan12343@gmail.com> wrote:
Is the script being run in parallel? If so, does it crash if you run it on only one core?
Nathan
On Monday, October 17, 2016, tazkera haque <h.tazkera@gmail.com> wrote:
HI people,
I am using yt 3.3.1 and submitting my SLURM script to stampede. I was using this script to find abundances of C, O etc through checkpoint files in FLASH. while my script worked fine with old yt (3.1) , suddenly it crashed today and returned me the following error:
*yt : [INFO ] 2016-10-17 23:22:24,295 Parameters: current_time = 28.1847530806* *yt : [INFO ] 2016-10-17 23:22:24,295 Parameters: domain_dimensions = [128 128 128]* *yt : [INFO ] 2016-10-17 23:22:24,296 Parameters: domain_left_edge = [ -2.80000000e+10 -2.80000000e+10 -2.80000000e+10]* *yt : [INFO ] 2016-10-17 23:22:24,296 Parameters: domain_right_edge = [ 2.80000000e+10 2.80000000e+10 2.80000000e+10]* *yt : [INFO ] 2016-10-17 23:22:24,296 Parameters: cosmological_simulation = 0.0* *Executin lessg abundance.py* *Traceback (most recent call last):* * File "abundance2.py", line 304, in <module>* * main(chkFilenames_own)* * File "abundance2.py", line 59, in main* * pf = yt.load(filenames[n])* * File "/work/03858/thaque56/sw/yt-new-3.3/yt-conda/lib/python2.7/site-packages/yt/convenience.py", line 79, in load* * if c._is_valid(*args, **kwargs): candidates.append(n)* * File "/work/03858/thaque56/sw/yt-new-3.3/yt-conda/lib/python2.7/site-packages/yt/frontends/flash/data_structures.py", line 478, in _is_valid* * if "bounding box" not in fileh["/"].keys() \* * File "/work/03858/thaque56/sw/yt-new-3.3/yt-conda/lib/python2.7/site-packages/h5py/_hl/base.py", line 368, in keys* * return list(self)* * File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper (/home/ilan/minonda/conda-bld/work/h5py/_objects.c:2696)* * File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper (/home/ilan/minonda/conda-bld/work/h5py/_objects.c:2654)* * File "/work/03858/thaque56/sw/yt-new-3.3/yt-conda/lib/python2.7/site-packages/h5py/_hl/group.py", line 303, in __len__* * return self.id.get_num_objs()* * File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper (/home/ilan/minonda/conda-bld/work/h5py/_objects.c:2696)* * File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper (/home/ilan/minonda/conda-bld/work/h5py/_objects.c:2654)* * File "h5py/h5g.pyx", line 321, in h5py.h5g.GroupID.get_num_objs (/home/ilan/minonda/conda-bld/work/h5py/h5g.c:4194)* *RuntimeError: Can't determine (Bad symbol table node signature)* *[c560-102.stampede.tacc.utexas.edu:mpispawn_0][child_handler] MPI process (rank: 0, pid: 22137) exited with status 1* *TACC: MPI job exited with code: 1*
*TACC: Shutdown complete. Exiting.*
I was wondering if there is something wrong with my code or the new yt. I am also attaching my code here to look at. Thanks in advance
Best Tazkera
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
I typed $ module avail HDF5 ------------------------------------------------- /opt/apps/intel15/mvapich2_2_1/modulefiles -------------------------------------------------- phdf5/1.8.16 -------------------------------------------------------- /opt/apps/intel15/modulefiles -------------------------------------------------------- hdf5/1.8.16 (m,L) Where: L: Module is loaded m: built for host and native MIC $ module avail h5py returns blank, if that helps On Tue, Oct 18, 2016 at 4:08 PM, Nathan Goldbaum <nathan12343@gmail.com> wrote:
Hi Tazkera,
When I tried googling your error message last night, I found that it's associated with more than one MPI process trying to access an HDF5 file at the same time, presumably using a serial version of the HDF5 library.
I just did a quick test and I'm unable to reproduce here on my laptop. Unfortunately I don't have access to stampede so can't reproduce there.
Can you share exactly which h5py and HDF5 library versions you're using?
-Nathan
On Tue, Oct 18, 2016 at 3:05 PM, tazkera haque <h.tazkera@gmail.com> wrote:
HI Nathan,
Sorry to bother you again, but the problem seems to prevail even working from $WORK directory. I got the same error msg this morning again with a different script. Do you see anything wrong with the script I attached ? I have used it for a long time now without any sort of error.
Thanks
On Tue, Oct 18, 2016 at 2:34 AM, tazkera haque <h.tazkera@gmail.com> wrote:
HI Nathan,
I figured out what was going wrong, I submitted my job script from the $SCRATCH folder. apparently jobs can only be submitted through the $WORK folder on Stampede. thanks for your prompt response though
On Tue, Oct 18, 2016 at 1:22 AM, tazkera haque <h.tazkera@gmail.com> wrote:
Hi Nathan,
I tried with one file in my ipython notebook, it seems to work there
On Tue, Oct 18, 2016 at 12:54 AM, tazkera haque <h.tazkera@gmail.com> wrote:
yes it's being run in parallel, I didn't check with one core yet, I will let you know what happens then
On Tue, Oct 18, 2016 at 12:50 AM, Nathan Goldbaum < nathan12343@gmail.com> wrote:
Is the script being run in parallel? If so, does it crash if you run it on only one core?
Nathan
On Monday, October 17, 2016, tazkera haque <h.tazkera@gmail.com> wrote:
> HI people, > > I am using yt 3.3.1 and submitting my SLURM script to stampede. > I was using this script to find abundances of C, O etc through > checkpoint files in FLASH. while my script worked fine with old yt (3.1) , > suddenly it crashed today and returned me the following error: > > *yt : [INFO ] 2016-10-17 23:22:24,295 Parameters: current_time > = 28.1847530806* > *yt : [INFO ] 2016-10-17 23:22:24,295 Parameters: > domain_dimensions = [128 128 128]* > *yt : [INFO ] 2016-10-17 23:22:24,296 Parameters: > domain_left_edge = [ -2.80000000e+10 -2.80000000e+10 > -2.80000000e+10]* > *yt : [INFO ] 2016-10-17 23:22:24,296 Parameters: > domain_right_edge = [ 2.80000000e+10 2.80000000e+10 > 2.80000000e+10]* > *yt : [INFO ] 2016-10-17 23:22:24,296 Parameters: > cosmological_simulation = 0.0* > *Executin lessg abundance.py* > *Traceback (most recent call last):* > * File "abundance2.py", line 304, in <module>* > * main(chkFilenames_own)* > * File "abundance2.py", line 59, in main* > * pf = yt.load(filenames[n])* > * File > "/work/03858/thaque56/sw/yt-new-3.3/yt-conda/lib/python2.7/site-packages/yt/convenience.py", > line 79, in load* > * if c._is_valid(*args, **kwargs): candidates.append(n)* > * File > "/work/03858/thaque56/sw/yt-new-3.3/yt-conda/lib/python2.7/site-packages/yt/frontends/flash/data_structures.py", > line 478, in _is_valid* > * if "bounding box" not in fileh["/"].keys() \* > * File > "/work/03858/thaque56/sw/yt-new-3.3/yt-conda/lib/python2.7/site-packages/h5py/_hl/base.py", > line 368, in keys* > * return list(self)* > * File "h5py/_objects.pyx", line 54, in > h5py._objects.with_phil.wrapper > (/home/ilan/minonda/conda-bld/work/h5py/_objects.c:2696)* > * File "h5py/_objects.pyx", line 55, in > h5py._objects.with_phil.wrapper > (/home/ilan/minonda/conda-bld/work/h5py/_objects.c:2654)* > * File > "/work/03858/thaque56/sw/yt-new-3.3/yt-conda/lib/python2.7/site-packages/h5py/_hl/group.py", > line 303, in __len__* > * return self.id.get_num_objs()* > * File "h5py/_objects.pyx", line 54, in > h5py._objects.with_phil.wrapper > (/home/ilan/minonda/conda-bld/work/h5py/_objects.c:2696)* > * File "h5py/_objects.pyx", line 55, in > h5py._objects.with_phil.wrapper > (/home/ilan/minonda/conda-bld/work/h5py/_objects.c:2654)* > * File "h5py/h5g.pyx", line 321, in h5py.h5g.GroupID.get_num_objs > (/home/ilan/minonda/conda-bld/work/h5py/h5g.c:4194)* > *RuntimeError: Can't determine (Bad symbol table node signature)* > *[c560-102.stampede.tacc.utexas.edu:mpispawn_0][child_handler] MPI > process (rank: 0, pid: 22137) exited with status 1* > *TACC: MPI job exited with code: 1* > > *TACC: Shutdown complete. Exiting.* > > I was wondering if there is something wrong with my code or the new > yt. I am also attaching my code here to look at. Thanks in advance > > Best > Tazkera >
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
Hi Tazkera, I have the access to Stampede but I can not reproduce your error either (using a slightly different script though).
apparently jobs can only be submitted through the $WORK folder on Stampede
Btw, I believe you could submit job from both work and scratch directory on Stampede. Could you run your code in interactive mode, by srun -p normal -t 0:30:00 -n 16 -A _your_project_code_ --pty /bin/bash -l and then enter python abundance.py and see what’s going on? Best wishes, -- Suoqing JI Ph.D Candidate Department of Physics University of California, Santa Barbara http://web.physics.ucsb.edu/~suoqing
On Oct 18, 2016, at 1:20 PM, tazkera haque <h.tazkera@gmail.com> wrote:
I typed
$ module avail HDF5
------------------------------------------------- /opt/apps/intel15/mvapich2_2_1/modulefiles -------------------------------------------------- phdf5/1.8.16
-------------------------------------------------------- /opt/apps/intel15/modulefiles -------------------------------------------------------- hdf5/1.8.16 (m,L)
Where: L: Module is loaded m: built for host and native MIC
$ module avail h5py returns blank, if that helps
On Tue, Oct 18, 2016 at 4:08 PM, Nathan Goldbaum <nathan12343@gmail.com <mailto:nathan12343@gmail.com>> wrote: Hi Tazkera,
When I tried googling your error message last night, I found that it's associated with more than one MPI process trying to access an HDF5 file at the same time, presumably using a serial version of the HDF5 library.
I just did a quick test and I'm unable to reproduce here on my laptop. Unfortunately I don't have access to stampede so can't reproduce there.
Can you share exactly which h5py and HDF5 library versions you're using?
-Nathan
On Tue, Oct 18, 2016 at 3:05 PM, tazkera haque <h.tazkera@gmail.com <mailto:h.tazkera@gmail.com>> wrote: HI Nathan,
Sorry to bother you again, but the problem seems to prevail even working from $WORK directory. I got the same error msg this morning again with a different script. Do you see anything wrong with the script I attached ? I have used it for a long time now without any sort of error.
Thanks
On Tue, Oct 18, 2016 at 2:34 AM, tazkera haque <h.tazkera@gmail.com <mailto:h.tazkera@gmail.com>> wrote: HI Nathan,
I figured out what was going wrong, I submitted my job script from the $SCRATCH folder. apparently jobs can only be submitted through the $WORK folder on Stampede. thanks for your prompt response though
On Tue, Oct 18, 2016 at 1:22 AM, tazkera haque <h.tazkera@gmail.com <mailto:h.tazkera@gmail.com>> wrote: Hi Nathan,
I tried with one file in my ipython notebook, it seems to work there
On Tue, Oct 18, 2016 at 12:54 AM, tazkera haque <h.tazkera@gmail.com <mailto:h.tazkera@gmail.com>> wrote: yes it's being run in parallel, I didn't check with one core yet, I will let you know what happens then
On Tue, Oct 18, 2016 at 12:50 AM, Nathan Goldbaum <nathan12343@gmail.com <mailto:nathan12343@gmail.com>> wrote: Is the script being run in parallel? If so, does it crash if you run it on only one core?
Nathan
On Monday, October 17, 2016, tazkera haque <h.tazkera@gmail.com <mailto:h.tazkera@gmail.com>> wrote: HI people,
I am using yt 3.3.1 and submitting my SLURM script to stampede. I was using this script to find abundances of C, O etc through checkpoint files in FLASH. while my script worked fine with old yt (3.1) , suddenly it crashed today and returned me the following error:
yt : [INFO ] 2016-10-17 23:22:24,295 Parameters: current_time = 28.1847530806 yt : [INFO ] 2016-10-17 23:22:24,295 Parameters: domain_dimensions = [128 128 128] yt : [INFO ] 2016-10-17 23:22:24,296 Parameters: domain_left_edge = [ -2.80000000e+10 -2.80000000e+10 -2.80000000e+10] yt : [INFO ] 2016-10-17 23:22:24,296 Parameters: domain_right_edge = [ 2.80000000e+10 2.80000000e+10 2.80000000e+10] yt : [INFO ] 2016-10-17 23:22:24,296 Parameters: cosmological_simulation = 0.0 Executin lessg abundance.py Traceback (most recent call last): File "abundance2.py", line 304, in <module> main(chkFilenames_own) File "abundance2.py", line 59, in main pf = yt.load(filenames[n]) File "/work/03858/thaque56/sw/yt-new-3.3/yt-conda/lib/python2.7/site-packages/yt/convenience.py", line 79, in load if c._is_valid(*args, **kwargs): candidates.append(n) File "/work/03858/thaque56/sw/yt-new-3.3/yt-conda/lib/python2.7/site-packages/yt/frontends/flash/data_structures.py", line 478, in _is_valid if "bounding box" not in fileh["/"].keys() \ File "/work/03858/thaque56/sw/yt-new-3.3/yt-conda/lib/python2.7/site-packages/h5py/_hl/base.py", line 368, in keys return list(self) File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper (/home/ilan/minonda/conda-bld/work/h5py/_objects.c:2696) File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper (/home/ilan/minonda/conda-bld/work/h5py/_objects.c:2654) File "/work/03858/thaque56/sw/yt-new-3.3/yt-conda/lib/python2.7/site-packages/h5py/_hl/group.py", line 303, in __len__ return self.id.get_num_objs() File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper (/home/ilan/minonda/conda-bld/work/h5py/_objects.c:2696) File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper (/home/ilan/minonda/conda-bld/work/h5py/_objects.c:2654) File "h5py/h5g.pyx", line 321, in h5py.h5g.GroupID.get_num_objs (/home/ilan/minonda/conda-bld/work/h5py/h5g.c:4194) RuntimeError: Can't determine (Bad symbol table node signature) [c560-102.stampede.tacc.utexas.edu:mpispawn_0][child_handler] MPI process (rank: 0, pid: 22137) exited with status 1 TACC: MPI job exited with code: 1
TACC: Shutdown complete. Exiting.
I was wondering if there is something wrong with my code or the new yt. I am also attaching my code here to look at. Thanks in advance
Best Tazkera
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org <mailto:yt-users@lists.spacepope.org> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org <http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org>
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org <mailto:yt-users@lists.spacepope.org> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org <http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org>
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org <mailto:yt-users@lists.spacepope.org> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org <http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org>
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
Hi Suoqing, I ran the code in the interactive mode and it returned me the same error as it was with submitting the job script. What happens is that It checks through all the files and instead of producing the output files or images, it returns me the error. Best On Tue, Oct 18, 2016 at 4:48 PM, Suoqing Ji <suoqing@physics.ucsb.edu> wrote:
Hi Tazkera,
I have the access to Stampede but I can not reproduce your error either (using a slightly different script though).
apparently jobs can only be submitted through the $WORK folder on Stampede
Btw, I believe you could submit job from both work and scratch directory on Stampede.
Could you run your code in interactive mode, by
srun -p normal -t 0:30:00 -n 16 -A _your_project_code_ --pty /bin/bash -l
and then enter
python abundance.py
and see what’s going on?
Best wishes, -- Suoqing JI Ph.D Candidate Department of Physics University of California, Santa Barbara http://web.physics.ucsb.edu/~suoqing
On Oct 18, 2016, at 1:20 PM, tazkera haque <h.tazkera@gmail.com> wrote:
I typed
$ module avail HDF5
------------------------------------------------- /opt/apps/intel15/mvapich2_2_1/modulefiles ------------------------------ -------------------- phdf5/1.8.16
-------------------------------------------------------- /opt/apps/intel15/modulefiles ------------------------------ -------------------------- hdf5/1.8.16 (m,L)
Where: L: Module is loaded m: built for host and native MIC
$ module avail h5py returns blank, if that helps
On Tue, Oct 18, 2016 at 4:08 PM, Nathan Goldbaum <nathan12343@gmail.com> wrote:
Hi Tazkera,
When I tried googling your error message last night, I found that it's associated with more than one MPI process trying to access an HDF5 file at the same time, presumably using a serial version of the HDF5 library.
I just did a quick test and I'm unable to reproduce here on my laptop. Unfortunately I don't have access to stampede so can't reproduce there.
Can you share exactly which h5py and HDF5 library versions you're using?
-Nathan
On Tue, Oct 18, 2016 at 3:05 PM, tazkera haque <h.tazkera@gmail.com> wrote:
HI Nathan,
Sorry to bother you again, but the problem seems to prevail even working from $WORK directory. I got the same error msg this morning again with a different script. Do you see anything wrong with the script I attached ? I have used it for a long time now without any sort of error.
Thanks
On Tue, Oct 18, 2016 at 2:34 AM, tazkera haque <h.tazkera@gmail.com> wrote:
HI Nathan,
I figured out what was going wrong, I submitted my job script from the $SCRATCH folder. apparently jobs can only be submitted through the $WORK folder on Stampede. thanks for your prompt response though
On Tue, Oct 18, 2016 at 1:22 AM, tazkera haque <h.tazkera@gmail.com> wrote:
Hi Nathan,
I tried with one file in my ipython notebook, it seems to work there
On Tue, Oct 18, 2016 at 12:54 AM, tazkera haque <h.tazkera@gmail.com> wrote:
yes it's being run in parallel, I didn't check with one core yet, I will let you know what happens then
On Tue, Oct 18, 2016 at 12:50 AM, Nathan Goldbaum < nathan12343@gmail.com> wrote:
> Is the script being run in parallel? If so, does it crash if you run > it on only one core? > > Nathan > > > On Monday, October 17, 2016, tazkera haque <h.tazkera@gmail.com> > wrote: > >> HI people, >> >> I am using yt 3.3.1 and submitting my SLURM script to stampede. >> I was using this script to find abundances of C, O etc through >> checkpoint files in FLASH. while my script worked fine with old yt (3.1) , >> suddenly it crashed today and returned me the following error: >> >> *yt : [INFO ] 2016-10-17 23:22:24,295 Parameters: current_time >> = 28.1847530806* >> *yt : [INFO ] 2016-10-17 23:22:24,295 Parameters: >> domain_dimensions = [128 128 128]* >> *yt : [INFO ] 2016-10-17 23:22:24,296 Parameters: >> domain_left_edge = [ -2.80000000e+10 -2.80000000e+10 >> -2.80000000e+10]* >> *yt : [INFO ] 2016-10-17 23:22:24,296 Parameters: >> domain_right_edge = [ 2.80000000e+10 2.80000000e+10 >> 2.80000000e+10]* >> *yt : [INFO ] 2016-10-17 23:22:24,296 Parameters: >> cosmological_simulation = 0.0* >> *Executin lessg abundance.py* >> *Traceback (most recent call last):* >> * File "abundance2.py", line 304, in <module>* >> * main(chkFilenames_own)* >> * File "abundance2.py", line 59, in main* >> * pf = yt.load(filenames[n])* >> * File >> "/work/03858/thaque56/sw/yt-new-3.3/yt-conda/lib/python2.7/site-packages/yt/convenience.py", >> line 79, in load* >> * if c._is_valid(*args, **kwargs): candidates.append(n)* >> * File >> "/work/03858/thaque56/sw/yt-new-3.3/yt-conda/lib/python2.7/site-packages/yt/frontends/flash/data_structures.py", >> line 478, in _is_valid* >> * if "bounding box" not in fileh["/"].keys() \* >> * File >> "/work/03858/thaque56/sw/yt-new-3.3/yt-conda/lib/python2.7/site-packages/h5py/_hl/base.py", >> line 368, in keys* >> * return list(self)* >> * File "h5py/_objects.pyx", line 54, in >> h5py._objects.with_phil.wrapper >> (/home/ilan/minonda/conda-bld/work/h5py/_objects.c:2696)* >> * File "h5py/_objects.pyx", line 55, in >> h5py._objects.with_phil.wrapper >> (/home/ilan/minonda/conda-bld/work/h5py/_objects.c:2654)* >> * File >> "/work/03858/thaque56/sw/yt-new-3.3/yt-conda/lib/python2.7/site-packages/h5py/_hl/group.py", >> line 303, in __len__* >> * return self.id.get_num_objs()* >> * File "h5py/_objects.pyx", line 54, in >> h5py._objects.with_phil.wrapper >> (/home/ilan/minonda/conda-bld/work/h5py/_objects.c:2696)* >> * File "h5py/_objects.pyx", line 55, in >> h5py._objects.with_phil.wrapper >> (/home/ilan/minonda/conda-bld/work/h5py/_objects.c:2654)* >> * File "h5py/h5g.pyx", line 321, in h5py.h5g.GroupID.get_num_objs >> (/home/ilan/minonda/conda-bld/work/h5py/h5g.c:4194)* >> *RuntimeError: Can't determine (Bad symbol table node signature)* >> *[c560-102.stampede.tacc.utexas.edu:mpispawn_0][child_handler] MPI >> process (rank: 0, pid: 22137) exited with status 1* >> *TACC: MPI job exited with code: 1* >> >> *TACC: Shutdown complete. Exiting.* >> >> I was wondering if there is something wrong with my code or the new >> yt. I am also attaching my code here to look at. Thanks in advance >> >> Best >> Tazkera >> > > _______________________________________________ > yt-users mailing list > yt-users@lists.spacepope.org > http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org > >
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
participants (3)
-
Nathan Goldbaum -
Suoqing Ji -
tazkera haque