Hello, I have a request for help. I am trying to run some analysis code remotely through ssh in yt on NASA Pleiades. I have been able to install yt and trident and am running in a virtualenv, but something is having a very deeply buried error that I don't understand and fatally kills any task I send to Pleiades. The same code works fine on my home laptop with the same input files, so I assume the problem is something to do with the yt distribution on Pleiades, but I used "pip install -U yt" so it should be up-to-date. The error code comes from the fortran_utils file, and it says "unpack requires a string argument of length 4" (myenv) cstrawn@pfe27:~/quasarlines/quasarlines> python quasar_scan.py VELA_v2_08 10MpcBox_csf512_a0.330.d "[H I,H II,O IV,O V,O VI,Si II,Si III,Si IV,C I,C II,C III,C IV,Mg II,Mg III,Mg IV]" yt : [INFO ] 2017-07-25 12:11:26,454 discovered particle_data:/home1/cstrawn/quasarlines/quasarlines/PMcrs0a0.330.DAT yt : [INFO ] 2017-07-25 12:11:26,454 discovered particle_header:/home1/cstrawn/quasarlines/quasarlines/PMcrda0.330.DAT yt : [INFO ] 2017-07-25 12:11:26,462 Using root level of 14 yt : [INFO ] 2017-07-25 12:11:26,463 Discovered 7 species of particles yt : [INFO ] 2017-07-25 12:11:26,463 Particle populations: 28016640 4263936 752640 123904 19552 2089406 85309 yt : [INFO ] 2017-07-25 12:11:26,658 Max level is 12 yt : [INFO ] 2017-07-25 12:11:26,682 Parameters: current_time = 3.350407555 Gyr yt : [INFO ] 2017-07-25 12:11:26,682 Parameters: domain_dimensions = [128 128 128] yt : [INFO ] 2017-07-25 12:11:26,683 Parameters: domain_left_edge = [ 0. 0. 0.] yt : [INFO ] 2017-07-25 12:11:26,683 Parameters: domain_right_edge = [ 1. 1. 1.] yt : [INFO ] 2017-07-25 12:11:26,683 Parameters: cosmological_simulation = True yt : [INFO ] 2017-07-25 12:11:26,683 Parameters: current_redshift = 2.02694081565 yt : [INFO ] 2017-07-25 12:11:26,683 Parameters: omega_lambda = 0.730000019073 yt : [INFO ] 2017-07-25 12:11:26,684 Parameters: omega_matter = 0.270000010729 yt : [INFO ] 2017-07-25 12:11:26,684 Parameters: hubble_constant = 0.699999988079 18426 LOSs to scan. Is this an appropriate number? (0 for no) 1 1/18426 Traceback (most recent call last): File "quasar_scan.py", line 375, in <module> q.get_coldens(save = 10) File "quasar_scan.py", line 148, in get_coldens ftype='gas') File "/u/cstrawn/myenv/lib/python2.7/site-packages/trident/ray_generator.py", line 239, in make_simple_ray particle_type = _determine_particle_type_from_ftype(ds, ftype) File "/u/cstrawn/myenv/lib/python2.7/site-packages/trident/ray_generator.py", line 590, in _determine_particle_type_from_ftype field_list_arr = np.asarray(ds.derived_field_list) File "/u/cstrawn/myenv/lib/python2.7/site-packages/yt/data_objects/static_output.py", line 164, in ireq self.index File "/u/cstrawn/myenv/lib/python2.7/site-packages/yt/data_objects/static_output.py", line 419, in index self, dataset_type=self.dataset_type) File "/u/cstrawn/myenv/lib/python2.7/site-packages/yt/frontends/art/data_structures.py", line 65, in __init__ super(ARTIndex, self).__init__(ds, dataset_type) File "/u/cstrawn/myenv/lib/python2.7/site-packages/yt/geometry/geometry_handler.py", line 50, in __init__ self._setup_geometry() File "/u/cstrawn/myenv/lib/python2.7/site-packages/yt/geometry/oct_geometry_handler.py", line 25, in _setup_geometry self._initialize_oct_handler() File "/u/cstrawn/myenv/lib/python2.7/site-packages/yt/frontends/art/data_structures.py", line 91, in _initialize_oct_handler self.domains] File "/u/cstrawn/myenv/lib/python2.7/site-packages/yt/frontends/art/data_structures.py", line 761, in level_count self.level_offsets File "/u/cstrawn/myenv/lib/python2.7/site-packages/yt/frontends/art/data_structures.py", line 782, in level_offsets self.ds.min_level, self.ds.max_level) File "/u/cstrawn/myenv/lib/python2.7/site-packages/yt/frontends/art/data_structures.py", line 820, in _count_art_octs ns = fpu.peek_record_size(f, endian='>') File "/u/cstrawn/myenv/lib/python2.7/site-packages/yt/utilities/fortran_utils.py", line 250, in peek_record_size s = struct.unpack('>i', f.read(struct.calcsize('>i'))) struct.error: unpack requires a string argument of length 4 Please let me know if this is a known problem or if you have any idea what I should do to avoid running into it. I don't really understand fortran enough to know what is even supposed to be happening in fortran_utils. Thanks so much, Clayton Strawn
Are you sure that the files are the same on your laptop and on Pleiades? Are you sure that you're running the same version of yt on your laptop and pleiades? A way to check the former would be to make a tarball containing the dataset and then find the tarball's SHA hash on your laptop and on Pleiades, e.g.: # on linux $ sha256sum my_data.tar # on MacOS $ shasum -a 256 my_data.tar If those commands don't print the same answer on your laptop and on Pleiades, that indicates that the files are not identical and perhaps there has been some corruption. That might be the source of the issue. You can check to make sure the yt versions are the same on your laptop and on Pleiades by running "yt version" on the bash command line. What's happening in the error you're running that yt is trying to parse the octree hierarchy in your data by reading some bytes from the data file. It's dying because it's getting back a piece of data that's not the shape it's expecting based on what we know about the ART data format. It's very odd that this works on your laptop using the exact same data file, because yt is making the same assumptions about the structure of the data file on Pleiades and your laptop. Finally, if none of the diagnostic suggestions I made above help, it would help to make further progress on this issue if you can share the data file that's triggering the problem so that one of us can reproduce the issue and debug locally, although I totally understand if there are possible concerns about sharing data. -Nathan On Tue, Jul 25, 2017 at 3:05 PM, Clayton Strawn <cjstrawn@ucsc.edu> wrote:
Hello,
I have a request for help.
I am trying to run some analysis code remotely through ssh in yt on NASA Pleiades. I have been able to install yt and trident and am running in a virtualenv, but something is having a very deeply buried error that I don't understand and fatally kills any task I send to Pleiades.
The same code works fine on my home laptop with the same input files, so I assume the problem is something to do with the yt distribution on Pleiades, but I used "pip install -U yt" so it should be up-to-date. The error code comes from the fortran_utils file, and it says "unpack requires a string argument of length 4"
(myenv) cstrawn@pfe27:~/quasarlines/quasarlines> python quasar_scan.py VELA_v2_08 10MpcBox_csf512_a0.330.d "[H I,H II,O IV,O V,O VI,Si II,Si III,Si IV,C I,C II,C III,C IV,Mg II,Mg III,Mg IV]" yt : [INFO ] 2017-07-25 12:11:26,454 discovered particle_data:/home1/cstrawn/quasarlines/quasarlines/PMcrs0a0.330.DAT yt : [INFO ] 2017-07-25 12:11:26,454 discovered particle_header:/home1/ cstrawn/quasarlines/quasarlines/PMcrda0.330.DAT yt : [INFO ] 2017-07-25 12:11:26,462 Using root level of 14 yt : [INFO ] 2017-07-25 12:11:26,463 Discovered 7 species of particles yt : [INFO ] 2017-07-25 12:11:26,463 Particle populations: 28016640 4263936 752640 123904 19552 2089406 85309 yt : [INFO ] 2017-07-25 12:11:26,658 Max level is 12 yt : [INFO ] 2017-07-25 12:11:26,682 Parameters: current_time = 3.350407555 Gyr yt : [INFO ] 2017-07-25 12:11:26,682 Parameters: domain_dimensions = [128 128 128] yt : [INFO ] 2017-07-25 12:11:26,683 Parameters: domain_left_edge = [ 0. 0. 0.] yt : [INFO ] 2017-07-25 12:11:26,683 Parameters: domain_right_edge = [ 1. 1. 1.] yt : [INFO ] 2017-07-25 12:11:26,683 Parameters: cosmological_simulation = True yt : [INFO ] 2017-07-25 12:11:26,683 Parameters: current_redshift = 2.02694081565 yt : [INFO ] 2017-07-25 12:11:26,683 Parameters: omega_lambda = 0.730000019073 yt : [INFO ] 2017-07-25 12:11:26,684 Parameters: omega_matter = 0.270000010729 yt : [INFO ] 2017-07-25 12:11:26,684 Parameters: hubble_constant = 0.699999988079 18426 LOSs to scan. Is this an appropriate number? (0 for no) 1 1/18426 Traceback (most recent call last): File "quasar_scan.py", line 375, in <module> q.get_coldens(save = 10) File "quasar_scan.py", line 148, in get_coldens ftype='gas') File "/u/cstrawn/myenv/lib/python2.7/site-packages/trident/ray_generator.py", line 239, in make_simple_ray particle_type = _determine_particle_type_from_ftype(ds, ftype) File "/u/cstrawn/myenv/lib/python2.7/site-packages/trident/ray_generator.py", line 590, in _determine_particle_type_from_ftype field_list_arr = np.asarray(ds.derived_field_list) File "/u/cstrawn/myenv/lib/python2.7/site-packages/yt/data_objects/static_output.py", line 164, in ireq self.index File "/u/cstrawn/myenv/lib/python2.7/site-packages/yt/data_objects/static_output.py", line 419, in index self, dataset_type=self.dataset_type) File "/u/cstrawn/myenv/lib/python2.7/site-packages/yt/frontends/art/data_structures.py", line 65, in __init__ super(ARTIndex, self).__init__(ds, dataset_type) File "/u/cstrawn/myenv/lib/python2.7/site-packages/yt/geometry/geometry_handler.py", line 50, in __init__ self._setup_geometry() File "/u/cstrawn/myenv/lib/python2.7/site-packages/yt/geometry/oct_geometry_handler.py", line 25, in _setup_geometry self._initialize_oct_handler() File "/u/cstrawn/myenv/lib/python2.7/site-packages/yt/frontends/art/data_structures.py", line 91, in _initialize_oct_handler self.domains] File "/u/cstrawn/myenv/lib/python2.7/site-packages/yt/frontends/art/data_structures.py", line 761, in level_count self.level_offsets File "/u/cstrawn/myenv/lib/python2.7/site-packages/yt/frontends/art/data_structures.py", line 782, in level_offsets self.ds.min_level, self.ds.max_level) File "/u/cstrawn/myenv/lib/python2.7/site-packages/yt/frontends/art/data_structures.py", line 820, in _count_art_octs ns = fpu.peek_record_size(f, endian='>') File "/u/cstrawn/myenv/lib/python2.7/site-packages/yt/utilities/fortran_utils.py", line 250, in peek_record_size s = struct.unpack('>i', f.read(struct.calcsize('>i'))) struct.error: unpack requires a string argument of length 4
Please let me know if this is a known problem or if you have any idea what I should do to avoid running into it. I don't really understand fortran enough to know what is even supposed to be happening in fortran_utils.
Thanks so much, Clayton Strawn
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
Well, I guess it wasn't exactly the same file. On my computer I have other galaxy snapshots from the same code, in the same format, which I assumed would be practically identical except for different numbers But when I tried porting it over to my computer it had the same error, and another file on Pleiades of a different snapshot worked fine. Apparently it was an error with the data storage/retrieval, and not with yt. So now it's working. Thanks so much for your quick response though! Clayton On Tue, Jul 25, 2017 at 1:15 PM, Nathan Goldbaum <nathan12343@gmail.com> wrote:
Are you sure that the files are the same on your laptop and on Pleiades? Are you sure that you're running the same version of yt on your laptop and pleiades?
A way to check the former would be to make a tarball containing the dataset and then find the tarball's SHA hash on your laptop and on Pleiades, e.g.:
# on linux $ sha256sum my_data.tar
# on MacOS $ shasum -a 256 my_data.tar
If those commands don't print the same answer on your laptop and on Pleiades, that indicates that the files are not identical and perhaps there has been some corruption. That might be the source of the issue.
You can check to make sure the yt versions are the same on your laptop and on Pleiades by running "yt version" on the bash command line.
What's happening in the error you're running that yt is trying to parse the octree hierarchy in your data by reading some bytes from the data file. It's dying because it's getting back a piece of data that's not the shape it's expecting based on what we know about the ART data format. It's very odd that this works on your laptop using the exact same data file, because yt is making the same assumptions about the structure of the data file on Pleiades and your laptop.
Finally, if none of the diagnostic suggestions I made above help, it would help to make further progress on this issue if you can share the data file that's triggering the problem so that one of us can reproduce the issue and debug locally, although I totally understand if there are possible concerns about sharing data.
-Nathan
On Tue, Jul 25, 2017 at 3:05 PM, Clayton Strawn <cjstrawn@ucsc.edu> wrote:
Hello,
I have a request for help.
I am trying to run some analysis code remotely through ssh in yt on NASA Pleiades. I have been able to install yt and trident and am running in a virtualenv, but something is having a very deeply buried error that I don't understand and fatally kills any task I send to Pleiades.
The same code works fine on my home laptop with the same input files, so I assume the problem is something to do with the yt distribution on Pleiades, but I used "pip install -U yt" so it should be up-to-date. The error code comes from the fortran_utils file, and it says "unpack requires a string argument of length 4"
(myenv) cstrawn@pfe27:~/quasarlines/quasarlines> python quasar_scan.py VELA_v2_08 10MpcBox_csf512_a0.330.d "[H I,H II,O IV,O V,O VI,Si II,Si III,Si IV,C I,C II,C III,C IV,Mg II,Mg III,Mg IV]" yt : [INFO ] 2017-07-25 12:11:26,454 discovered particle_data:/home1/cstrawn/quasarlines/quasarlines/PMcrs0a0.330.DAT yt : [INFO ] 2017-07-25 12:11:26,454 discovered particle_header:/home1/cstrawn/quasarlines/quasarlines/PMcrda0.330.DAT yt : [INFO ] 2017-07-25 12:11:26,462 Using root level of 14 yt : [INFO ] 2017-07-25 12:11:26,463 Discovered 7 species of particles yt : [INFO ] 2017-07-25 12:11:26,463 Particle populations: 28016640 4263936 752640 123904 19552 2089406 85309 yt : [INFO ] 2017-07-25 12:11:26,658 Max level is 12 yt : [INFO ] 2017-07-25 12:11:26,682 Parameters: current_time = 3.350407555 Gyr yt : [INFO ] 2017-07-25 12:11:26,682 Parameters: domain_dimensions = [128 128 128] yt : [INFO ] 2017-07-25 12:11:26,683 Parameters: domain_left_edge = [ 0. 0. 0.] yt : [INFO ] 2017-07-25 12:11:26,683 Parameters: domain_right_edge = [ 1. 1. 1.] yt : [INFO ] 2017-07-25 12:11:26,683 Parameters: cosmological_simulation = True yt : [INFO ] 2017-07-25 12:11:26,683 Parameters: current_redshift = 2.02694081565 yt : [INFO ] 2017-07-25 12:11:26,683 Parameters: omega_lambda = 0.730000019073 yt : [INFO ] 2017-07-25 12:11:26,684 Parameters: omega_matter = 0.270000010729 yt : [INFO ] 2017-07-25 12:11:26,684 Parameters: hubble_constant = 0.699999988079 18426 LOSs to scan. Is this an appropriate number? (0 for no) 1 1/18426 Traceback (most recent call last): File "quasar_scan.py", line 375, in <module> q.get_coldens(save = 10) File "quasar_scan.py", line 148, in get_coldens ftype='gas') File "/u/cstrawn/myenv/lib/python2.7/site-packages/trident/ray_generator.py", line 239, in make_simple_ray particle_type = _determine_particle_type_from_ftype(ds, ftype) File "/u/cstrawn/myenv/lib/python2.7/site-packages/trident/ray_generator.py", line 590, in _determine_particle_type_from_ftype field_list_arr = np.asarray(ds.derived_field_list) File "/u/cstrawn/myenv/lib/python2.7/site-packages/yt/data_objects/static_output.py", line 164, in ireq self.index File "/u/cstrawn/myenv/lib/python2.7/site-packages/yt/data_objects/static_output.py", line 419, in index self, dataset_type=self.dataset_type) File "/u/cstrawn/myenv/lib/python2.7/site-packages/yt/frontends/art/data_structures.py", line 65, in __init__ super(ARTIndex, self).__init__(ds, dataset_type) File "/u/cstrawn/myenv/lib/python2.7/site-packages/yt/geometry/geometry_handler.py", line 50, in __init__ self._setup_geometry() File "/u/cstrawn/myenv/lib/python2.7/site-packages/yt/geometry/oct_geometry_handler.py", line 25, in _setup_geometry self._initialize_oct_handler() File "/u/cstrawn/myenv/lib/python2.7/site-packages/yt/frontends/art/data_structures.py", line 91, in _initialize_oct_handler self.domains] File "/u/cstrawn/myenv/lib/python2.7/site-packages/yt/frontends/art/data_structures.py", line 761, in level_count self.level_offsets File "/u/cstrawn/myenv/lib/python2.7/site-packages/yt/frontends/art/data_structures.py", line 782, in level_offsets self.ds.min_level, self.ds.max_level) File "/u/cstrawn/myenv/lib/python2.7/site-packages/yt/frontends/art/data_structures.py", line 820, in _count_art_octs ns = fpu.peek_record_size(f, endian='>') File "/u/cstrawn/myenv/lib/python2.7/site-packages/yt/utilities/fortran_utils.py", line 250, in peek_record_size s = struct.unpack('>i', f.read(struct.calcsize('>i'))) struct.error: unpack requires a string argument of length 4
Please let me know if this is a known problem or if you have any idea what I should do to avoid running into it. I don't really understand fortran enough to know what is even supposed to be happening in fortran_utils.
Thanks so much, Clayton Strawn
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
participants (2)
-
Clayton Strawn -
Nathan Goldbaum