Well, I guess it wasn't exactly the same file. On my computer I have other galaxy snapshots from the same code, in the same format, which I assumed would be practically identical except for different numbers

But when I tried porting it over to my computer it had the same error, and another file on Pleiades of a different snapshot worked fine. Apparently it was an error with the data storage/retrieval, and not with yt. So now it's working.

Thanks so much for your quick response though! 

Clayton

On Tue, Jul 25, 2017 at 1:15 PM, Nathan Goldbaum <nathan12343@gmail.com> wrote:
Are you sure that the files are the same on your laptop and on Pleiades? Are you sure that you're running the same version of yt on your laptop and pleiades?

A way to check the former would be to make a tarball containing the dataset and then find the tarball's SHA hash on your laptop and on Pleiades, e.g.:

# on linux
$ sha256sum my_data.tar

# on MacOS
$ shasum -a 256 my_data.tar

If those commands don't print the same answer on your laptop and on Pleiades, that indicates that the files are not identical and perhaps there has been some corruption.  That might be the source of the issue.

You can check to make sure the yt versions are the same on your laptop and on Pleiades by running "yt version" on the bash command line.

What's happening in the error you're running that yt is trying to parse the octree hierarchy in your data by reading some bytes from the data file.  It's dying because it's getting back a piece of data that's not the shape it's expecting based on what we know about the ART data format. It's very odd that this works on your laptop using the exact same data file, because yt is making the same assumptions about the structure of the data file on Pleiades and your laptop.

Finally, if none of the diagnostic suggestions I made above help, it would help to make further progress on this issue if you can share the data file that's triggering the problem so that one of us can reproduce the issue and debug locally, although I totally understand if there are possible concerns about sharing data.

-Nathan

On Tue, Jul 25, 2017 at 3:05 PM, Clayton Strawn <cjstrawn@ucsc.edu> wrote:
Hello,

I have a request for help. 

I am trying to run some analysis code remotely through ssh in yt on NASA Pleiades. I have been able to install yt and trident and am running in a virtualenv, but something is having a very deeply buried error that I don't understand and fatally kills any task I send to Pleiades.

The same code works fine on my home laptop with the same input files, so I assume the problem is something to do with the yt distribution on Pleiades, but I used "pip install -U yt" so it should be up-to-date. The error code comes from the fortran_utils file, and it says "unpack requires a string argument of length 4"

(myenv) cstrawn@pfe27:~/quasarlines/quasarlines> python quasar_scan.py VELA_v2_08 10MpcBox_csf512_a0.330.d "[H I,H II,O IV,O V,O VI,Si II,Si III,Si IV,C I,C II,C III,C IV,Mg II,Mg III,Mg IV]"
yt : [INFO     ] 2017-07-25 12:11:26,454 discovered particle_data:/home1/cstrawn/quasarlines/quasarlines/PMcrs0a0.330.DAT
yt : [INFO     ] 2017-07-25 12:11:26,454 discovered particle_header:/home1/cstrawn/quasarlines/quasarlines/PMcrda0.330.DAT
yt : [INFO     ] 2017-07-25 12:11:26,462 Using root level of 14
yt : [INFO     ] 2017-07-25 12:11:26,463 Discovered 7 species of particles
yt : [INFO     ] 2017-07-25 12:11:26,463 Particle populations:  28016640   4263936    752640    123904     19552   2089406    85309
yt : [INFO     ] 2017-07-25 12:11:26,658 Max level is 12
yt : [INFO     ] 2017-07-25 12:11:26,682 Parameters: current_time              = 3.350407555 Gyr
yt : [INFO     ] 2017-07-25 12:11:26,682 Parameters: domain_dimensions         = [128 128 128]
yt : [INFO     ] 2017-07-25 12:11:26,683 Parameters: domain_left_edge          = [ 0.  0.  0.]
yt : [INFO     ] 2017-07-25 12:11:26,683 Parameters: domain_right_edge         = [ 1.  1.  1.]
yt : [INFO     ] 2017-07-25 12:11:26,683 Parameters: cosmological_simulation   = True
yt : [INFO     ] 2017-07-25 12:11:26,683 Parameters: current_redshift          = 2.02694081565
yt : [INFO     ] 2017-07-25 12:11:26,683 Parameters: omega_lambda              = 0.730000019073
yt : [INFO     ] 2017-07-25 12:11:26,684 Parameters: omega_matter              = 0.270000010729
yt : [INFO     ] 2017-07-25 12:11:26,684 Parameters: hubble_constant           = 0.699999988079
18426 LOSs to scan.
Is this an appropriate number? (0 for no)
1
1/18426
Traceback (most recent call last):
  File "quasar_scan.py", line 375, in <module>
    q.get_coldens(save = 10)
  File "quasar_scan.py", line 148, in get_coldens
    ftype='gas')
  File "/u/cstrawn/myenv/lib/python2.7/site-packages/trident/ray_generator.py", line 239, in make_simple_ray
    particle_type = _determine_particle_type_from_ftype(ds, ftype)
  File "/u/cstrawn/myenv/lib/python2.7/site-packages/trident/ray_generator.py", line 590, in _determine_particle_type_from_ftype
    field_list_arr = np.asarray(ds.derived_field_list)
  File "/u/cstrawn/myenv/lib/python2.7/site-packages/yt/data_objects/static_output.py", line 164, in ireq
    self.index
  File "/u/cstrawn/myenv/lib/python2.7/site-packages/yt/data_objects/static_output.py", line 419, in index
    self, dataset_type=self.dataset_type)
  File "/u/cstrawn/myenv/lib/python2.7/site-packages/yt/frontends/art/data_structures.py", line 65, in __init__
    super(ARTIndex, self).__init__(ds, dataset_type)
  File "/u/cstrawn/myenv/lib/python2.7/site-packages/yt/geometry/geometry_handler.py", line 50, in __init__
    self._setup_geometry()
  File "/u/cstrawn/myenv/lib/python2.7/site-packages/yt/geometry/oct_geometry_handler.py", line 25, in _setup_geometry
    self._initialize_oct_handler()
  File "/u/cstrawn/myenv/lib/python2.7/site-packages/yt/frontends/art/data_structures.py", line 91, in _initialize_oct_handler
    self.domains]
  File "/u/cstrawn/myenv/lib/python2.7/site-packages/yt/frontends/art/data_structures.py", line 761, in level_count
    self.level_offsets
  File "/u/cstrawn/myenv/lib/python2.7/site-packages/yt/frontends/art/data_structures.py", line 782, in level_offsets
    self.ds.min_level, self.ds.max_level)
  File "/u/cstrawn/myenv/lib/python2.7/site-packages/yt/frontends/art/data_structures.py", line 820, in _count_art_octs
    ns = fpu.peek_record_size(f, endian='>')
  File "/u/cstrawn/myenv/lib/python2.7/site-packages/yt/utilities/fortran_utils.py", line 250, in peek_record_size
    s = struct.unpack('>i', f.read(struct.calcsize('>i')))
struct.error: unpack requires a string argument of length 4


Please let me know if this is a known problem or if you have any idea what I should do to avoid running into it. I don't really understand fortran enough to know what is even supposed to be happening in fortran_utils.

Thanks so much,
Clayton Strawn


_______________________________________________
yt-users mailing list
yt-users@lists.spacepope.org
http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org



_______________________________________________
yt-users mailing list
yt-users@lists.spacepope.org
http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org