Problem loading FLASH2.5 data
Dear YT users, I upgraded my version of YT from an older version that still installed the libraries on its own (version downloaded 2016/06/21) to a more recent version using yt-conda. With the new version I cannot read old FLASH2.5 data any more. The previous version of yt gave me the correct output yt : [INFO ] 2016-09-09 16:37:59,285 Parameters: current_time = 1260000000.0 yt : [INFO ] 2016-09-09 16:37:59,286 Parameters: domain_dimensions = [8 8 8] yt : [INFO ] 2016-09-09 16:37:59,289 Parameters: domain_left_edge = [ -4.00000000e+17 -4.00000000e+17 -4.00000000e+17] yt : [INFO ] 2016-09-09 16:37:59,290 Parameters: domain_right_edge = [ 4.00000000e+17 4.00000000e+17 4.00000000e+17] yt : [INFO ] 2016-09-09 16:37:59,291 Parameters: cosmological_simulation = 0.0 the new version does not read some of the numbers properly, yt : [INFO ] 2016-09-09 16:39:11,692 Parameters: current_time = 1.06099789548e-313 yt : [INFO ] 2016-09-09 16:39:11,692 Parameters: domain_dimensions = [1104332420 0 1102957925] yt : [INFO ] 2016-09-09 16:39:11,693 Parameters: domain_left_edge = [ -4.00000000e+17 -4.00000000e+17 -4.00000000e+17] yt : [INFO ] 2016-09-09 16:39:11,694 Parameters: domain_right_edge = [ 4.00000000e+17 4.00000000e+17 4.00000000e+17] yt : [INFO ] 2016-09-09 16:39:11,695 Parameters: cosmological_simulation = 0.0 The time value as well as the dimensions are messed up and thus yt crashes with a memory error at some point. To me it looks like a conversion problem between big and little endian or some similar conversion problem between single and double precision. I tried various combinations of MAC and linux and the problem does not seem to operating system dependent. Quite a few functions have changed in terms of reading particle data, but the simulation does not contain particles. I tried to find the part of the code that reads the domain_dimensions and looked at file yt-conda/lib/python2.7/site-packages/yt/frontends/flash/data_structures.py where in line 99ff the block dimensions are read. But this part of the code is not executed (checked with print statements in iyt). Loading the hdf5 file with h5py and using the code in line 104 gives me the correct numbers, but I guess yt reads the data from somewhere else? Any hints on how the hierarchy of reading data is organized or where to search? I know FLASH2.5 is very old but we still have a lot of (quite expensive) F2.5 data. With best regards, Philipp Girichidis
Hi Phillip, I don't think we have any Flash2.5 test data on yt-project.org/data. We use the datasets there in our test suite, so if we don't have any such data that means that our support for a data format isn't tested and is subject to breakage. Would you be willing to share some test data that the yt project can use for debugging purposes? It should be relatively small, should trigger the error you're seeing, and should be ok to host publicly on yt-project.org/data and use in our test suite. If so, we can look into restoring support and making sure we don't regress again. You can share the data using e.g. Dropbox, google drive, or by using the yt curldrop: https://docs.hub.yt/services.html#curldrop Nathan On Tuesday, October 18, 2016, Philipp Girichidis <keule-phg@gmx.de> wrote:
Dear YT users,
I upgraded my version of YT from an older version that still installed the libraries on its own (version downloaded 2016/06/21) to a more recent version using yt-conda. With the new version I cannot read old FLASH2.5 data any more. The previous version of yt gave me the correct output
yt : [INFO ] 2016-09-09 16:37:59,285 Parameters: current_time = 1260000000.0 yt : [INFO ] 2016-09-09 16:37:59,286 Parameters: domain_dimensions = [8 8 8] yt : [INFO ] 2016-09-09 16:37:59,289 Parameters: domain_left_edge = [ -4.00000000e+17 -4.00000000e+17 -4.00000000e+17] yt : [INFO ] 2016-09-09 16:37:59,290 Parameters: domain_right_edge = [ 4.00000000e+17 4.00000000e+17 4.00000000e+17] yt : [INFO ] 2016-09-09 16:37:59,291 Parameters: cosmological_simulation = 0.0
the new version does not read some of the numbers properly,
yt : [INFO ] 2016-09-09 16:39:11,692 Parameters: current_time = 1.06099789548e-313 yt : [INFO ] 2016-09-09 16:39:11,692 Parameters: domain_dimensions = [1104332420 0 1102957925] yt : [INFO ] 2016-09-09 16:39:11,693 Parameters: domain_left_edge = [ -4.00000000e+17 -4.00000000e+17 -4.00000000e+17] yt : [INFO ] 2016-09-09 16:39:11,694 Parameters: domain_right_edge = [ 4.00000000e+17 4.00000000e+17 4.00000000e+17] yt : [INFO ] 2016-09-09 16:39:11,695 Parameters: cosmological_simulation = 0.0
The time value as well as the dimensions are messed up and thus yt crashes with a memory error at some point. To me it looks like a conversion problem between big and little endian or some similar conversion problem between single and double precision. I tried various combinations of MAC and linux and the problem does not seem to operating system dependent.
Quite a few functions have changed in terms of reading particle data, but the simulation does not contain particles. I tried to find the part of the code that reads the domain_dimensions and looked at file yt-conda/lib/python2.7/site-packages/yt/frontends/flash/data_structures.py where in line 99ff the block dimensions are read. But this part of the code is not executed (checked with print statements in iyt). Loading the hdf5 file with h5py and using the code in line 104 gives me the correct numbers, but I guess yt reads the data from somewhere else?
Any hints on how the hierarchy of reading data is organized or where to search? I know FLASH2.5 is very old but we still have a lot of (quite expensive) F2.5 data.
With best regards, Philipp Girichidis _______________________________________________ yt-users mailing list yt-users@lists.spacepope.org <javascript:;> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
Hi Nathan, thank you for the fast reply. Sorry, I didn’t check the sample data for F2.5 files. I uploaded a small test file here: http://use.yt/upload/b895e885 <http://use.yt/upload/b895e885> which is at time 0 (maybe not a very clever idea if the time is read incorrectly) and another file (http://use.yt/upload/1106d6c4 <http://use.yt/upload/1106d6c4>) at time 23.8 kyr. The block size is [8,8,8]. With best regards, Philipp
On 18 Oct 2016, at 15:37, Nathan Goldbaum <nathan12343@gmail.com> wrote:
Hi Phillip,
I don't think we have any Flash2.5 test data on yt-project.org/data <http://yt-project.org/data>. We use the datasets there in our test suite, so if we don't have any such data that means that our support for a data format isn't tested and is subject to breakage.
Would you be willing to share some test data that the yt project can use for debugging purposes? It should be relatively small, should trigger the error you're seeing, and should be ok to host publicly on yt-project.org/data <http://yt-project.org/data> and use in our test suite. If so, we can look into restoring support and making sure we don't regress again.
You can share the data using e.g. Dropbox, google drive, or by using the yt curldrop:
https://docs.hub.yt/services.html#curldrop <https://docs.hub.yt/services.html#curldrop>
Nathan
On Tuesday, October 18, 2016, Philipp Girichidis <keule-phg@gmx.de <mailto:keule-phg@gmx.de>> wrote: Dear YT users,
I upgraded my version of YT from an older version that still installed the libraries on its own (version downloaded 2016/06/21) to a more recent version using yt-conda. With the new version I cannot read old FLASH2.5 data any more. The previous version of yt gave me the correct output
yt : [INFO ] 2016-09-09 16:37:59,285 Parameters: current_time = 1260000000.0 yt : [INFO ] 2016-09-09 16:37:59,286 Parameters: domain_dimensions = [8 8 8] yt : [INFO ] 2016-09-09 16:37:59,289 Parameters: domain_left_edge = [ -4.00000000e+17 -4.00000000e+17 -4.00000000e+17] yt : [INFO ] 2016-09-09 16:37:59,290 Parameters: domain_right_edge = [ 4.00000000e+17 4.00000000e+17 4.00000000e+17] yt : [INFO ] 2016-09-09 16:37:59,291 Parameters: cosmological_simulation = 0.0
the new version does not read some of the numbers properly,
yt : [INFO ] 2016-09-09 16:39:11,692 Parameters: current_time = 1.06099789548e-313 yt : [INFO ] 2016-09-09 16:39:11,692 Parameters: domain_dimensions = [1104332420 0 1102957925] yt : [INFO ] 2016-09-09 16:39:11,693 Parameters: domain_left_edge = [ -4.00000000e+17 -4.00000000e+17 -4.00000000e+17] yt : [INFO ] 2016-09-09 16:39:11,694 Parameters: domain_right_edge = [ 4.00000000e+17 4.00000000e+17 4.00000000e+17] yt : [INFO ] 2016-09-09 16:39:11,695 Parameters: cosmological_simulation = 0.0
The time value as well as the dimensions are messed up and thus yt crashes with a memory error at some point. To me it looks like a conversion problem between big and little endian or some similar conversion problem between single and double precision. I tried various combinations of MAC and linux and the problem does not seem to operating system dependent.
Quite a few functions have changed in terms of reading particle data, but the simulation does not contain particles. I tried to find the part of the code that reads the domain_dimensions and looked at file yt-conda/lib/python2.7/site-packages/yt/frontends/flash/data_structures.py where in line 99ff the block dimensions are read. But this part of the code is not executed (checked with print statements in iyt). Loading the hdf5 file with h5py and using the code in line 104 gives me the correct numbers, but I guess yt reads the data from somewhere else?
Any hints on how the hierarchy of reading data is organized or where to search? I know FLASH2.5 is very old but we still have a lot of (quite expensive) F2.5 data.
With best regards, Philipp Girichidis _______________________________________________ yt-users mailing list yt-users@lists.spacepope.org <javascript:;> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org <http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org> _______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
Hi Nathan and Phillip, Since I am nominally the FLASH guy and used to run 2.5 I can check this one out. ================================ John ZuHone, Chandra/ACIS Operations Harvard-Smithsonian Center for Astrophysics 60 Garden St., MS-67 (w) 617-496-1816 Cambridge, MA 02138. (m) 781-708-5004 john.zuhone@cfa.harvard.edu http://hea-www.cfa.harvard.edu/~jzuhone ================================
On Oct 18, 2016, at 10:11 AM, Philipp <keule-phg@gmx.de> wrote:
Hi Nathan,
thank you for the fast reply. Sorry, I didn’t check the sample data for F2.5 files. I uploaded a small test file here: http://use.yt/upload/b895e885 which is at time 0 (maybe not a very clever idea if the time is read incorrectly) and another file (http://use.yt/upload/1106d6c4) at time 23.8 kyr. The block size is [8,8,8].
With best regards, Philipp
On 18 Oct 2016, at 15:37, Nathan Goldbaum <nathan12343@gmail.com> wrote:
Hi Phillip,
I don't think we have any Flash2.5 test data on yt-project.org/data. We use the datasets there in our test suite, so if we don't have any such data that means that our support for a data format isn't tested and is subject to breakage.
Would you be willing to share some test data that the yt project can use for debugging purposes? It should be relatively small, should trigger the error you're seeing, and should be ok to host publicly on yt-project.org/data and use in our test suite. If so, we can look into restoring support and making sure we don't regress again.
You can share the data using e.g. Dropbox, google drive, or by using the yt curldrop:
https://docs.hub.yt/services.html#curldrop
Nathan
On Tuesday, October 18, 2016, Philipp Girichidis <keule-phg@gmx.de> wrote: Dear YT users,
I upgraded my version of YT from an older version that still installed the libraries on its own (version downloaded 2016/06/21) to a more recent version using yt-conda. With the new version I cannot read old FLASH2.5 data any more. The previous version of yt gave me the correct output
yt : [INFO ] 2016-09-09 16:37:59,285 Parameters: current_time = 1260000000.0 yt : [INFO ] 2016-09-09 16:37:59,286 Parameters: domain_dimensions = [8 8 8] yt : [INFO ] 2016-09-09 16:37:59,289 Parameters: domain_left_edge = [ -4.00000000e+17 -4.00000000e+17 -4.00000000e+17] yt : [INFO ] 2016-09-09 16:37:59,290 Parameters: domain_right_edge = [ 4.00000000e+17 4.00000000e+17 4.00000000e+17] yt : [INFO ] 2016-09-09 16:37:59,291 Parameters: cosmological_simulation = 0.0
the new version does not read some of the numbers properly,
yt : [INFO ] 2016-09-09 16:39:11,692 Parameters: current_time = 1.06099789548e-313 yt : [INFO ] 2016-09-09 16:39:11,692 Parameters: domain_dimensions = [1104332420 0 1102957925] yt : [INFO ] 2016-09-09 16:39:11,693 Parameters: domain_left_edge = [ -4.00000000e+17 -4.00000000e+17 -4.00000000e+17] yt : [INFO ] 2016-09-09 16:39:11,694 Parameters: domain_right_edge = [ 4.00000000e+17 4.00000000e+17 4.00000000e+17] yt : [INFO ] 2016-09-09 16:39:11,695 Parameters: cosmological_simulation = 0.0
The time value as well as the dimensions are messed up and thus yt crashes with a memory error at some point. To me it looks like a conversion problem between big and little endian or some similar conversion problem between single and double precision. I tried various combinations of MAC and linux and the problem does not seem to operating system dependent.
Quite a few functions have changed in terms of reading particle data, but the simulation does not contain particles. I tried to find the part of the code that reads the domain_dimensions and looked at file yt-conda/lib/python2.7/site-packages/yt/frontends/flash/data_structures.py where in line 99ff the block dimensions are read. But this part of the code is not executed (checked with print statements in iyt). Loading the hdf5 file with h5py and using the code in line 104 gives me the correct numbers, but I guess yt reads the data from somewhere else?
Any hints on how the hierarchy of reading data is organized or where to search? I know FLASH2.5 is very old but we still have a lot of (quite expensive) F2.5 data.
With best regards, Philipp Girichidis _______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
Hi John and Phillip, I’m actively using yt with both Flash 4 and 2.5 data files now with up to date yt (3.3.1) on both my Mac laptop and NASA’s pleiades data analysis nodes, installed via pip; I have no trouble reading them or the test files Phillip uploaded. Maybe try installing with pip instead of conda? This is the output I get for the _0015 test file: yt : [INFO ] 2016-10-18 11:43:55,739 Parameters: current_time = 751000000000.0 yt : [INFO ] 2016-10-18 11:43:55,739 Parameters: domain_dimensions = [8 8 8] yt : [INFO ] 2016-10-18 11:43:55,740 Parameters: domain_left_edge = [ -2.00000000e+18 -2.00000000e+18 -2.00000000e+18] yt : [INFO ] 2016-10-18 11:43:55,741 Parameters: domain_right_edge = [ 2.00000000e+18 2.00000000e+18 2.00000000e+18] yt : [INFO ] 2016-10-18 11:43:55,742 Parameters: cosmological_simulation = 0.0 Happy to help diagnose further. -Alex --------- Alex Hill Postdoctoral Research Associate Office: KINSC Link L106 Department of Astronomy, Haverford College 370 Lancaster Ave, Haverford, PA 19041 USA phone: +1 484 297 2136 email/iMessage: ashill@haverford.edu
On Oct 18, 2016, at 10:12, John ZuHone <jzuhone@gmail.com> wrote:
Hi Nathan and Phillip,
Since I am nominally the FLASH guy and used to run 2.5 I can check this one out.
================================ John ZuHone, Chandra/ACIS Operations Harvard-Smithsonian Center for Astrophysics
60 Garden St., MS-67 (w) 617-496-1816 Cambridge, MA 02138. (m) 781-708-5004
john.zuhone@cfa.harvard.edu http://hea-www.cfa.harvard.edu/~jzuhone ================================
On Oct 18, 2016, at 10:11 AM, Philipp <keule-phg@gmx.de> wrote:
Hi Nathan,
thank you for the fast reply. Sorry, I didn’t check the sample data for F2.5 files. I uploaded a small test file here: http://use.yt/upload/b895e885 which is at time 0 (maybe not a very clever idea if the time is read incorrectly) and another file (http://use.yt/upload/1106d6c4) at time 23.8 kyr. The block size is [8,8,8].
With best regards, Philipp
On 18 Oct 2016, at 15:37, Nathan Goldbaum <nathan12343@gmail.com> wrote:
Hi Phillip,
I don't think we have any Flash2.5 test data on yt-project.org/data. We use the datasets there in our test suite, so if we don't have any such data that means that our support for a data format isn't tested and is subject to breakage.
Would you be willing to share some test data that the yt project can use for debugging purposes? It should be relatively small, should trigger the error you're seeing, and should be ok to host publicly on yt-project.org/data and use in our test suite. If so, we can look into restoring support and making sure we don't regress again.
You can share the data using e.g. Dropbox, google drive, or by using the yt curldrop:
https://docs.hub.yt/services.html#curldrop
Nathan
On Tuesday, October 18, 2016, Philipp Girichidis <keule-phg@gmx.de> wrote: Dear YT users,
I upgraded my version of YT from an older version that still installed the libraries on its own (version downloaded 2016/06/21) to a more recent version using yt-conda. With the new version I cannot read old FLASH2.5 data any more. The previous version of yt gave me the correct output
yt : [INFO ] 2016-09-09 16:37:59,285 Parameters: current_time = 1260000000.0 yt : [INFO ] 2016-09-09 16:37:59,286 Parameters: domain_dimensions = [8 8 8] yt : [INFO ] 2016-09-09 16:37:59,289 Parameters: domain_left_edge = [ -4.00000000e+17 -4.00000000e+17 -4.00000000e+17] yt : [INFO ] 2016-09-09 16:37:59,290 Parameters: domain_right_edge = [ 4.00000000e+17 4.00000000e+17 4.00000000e+17] yt : [INFO ] 2016-09-09 16:37:59,291 Parameters: cosmological_simulation = 0.0
the new version does not read some of the numbers properly,
yt : [INFO ] 2016-09-09 16:39:11,692 Parameters: current_time = 1.06099789548e-313 yt : [INFO ] 2016-09-09 16:39:11,692 Parameters: domain_dimensions = [1104332420 0 1102957925] yt : [INFO ] 2016-09-09 16:39:11,693 Parameters: domain_left_edge = [ -4.00000000e+17 -4.00000000e+17 -4.00000000e+17] yt : [INFO ] 2016-09-09 16:39:11,694 Parameters: domain_right_edge = [ 4.00000000e+17 4.00000000e+17 4.00000000e+17] yt : [INFO ] 2016-09-09 16:39:11,695 Parameters: cosmological_simulation = 0.0
The time value as well as the dimensions are messed up and thus yt crashes with a memory error at some point. To me it looks like a conversion problem between big and little endian or some similar conversion problem between single and double precision. I tried various combinations of MAC and linux and the problem does not seem to operating system dependent.
Quite a few functions have changed in terms of reading particle data, but the simulation does not contain particles. I tried to find the part of the code that reads the domain_dimensions and looked at file yt-conda/lib/python2.7/site-packages/yt/frontends/flash/data_structures.py where in line 99ff the block dimensions are read. But this part of the code is not executed (checked with print statements in iyt). Loading the hdf5 file with h5py and using the code in line 104 gives me the correct numbers, but I guess yt reads the data from somewhere else?
Any hints on how the hierarchy of reading data is organized or where to search? I know FLASH2.5 is very old but we still have a lot of (quite expensive) F2.5 data.
With best regards, Philipp Girichidis _______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org _______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
Hey all, I can confirm the strange behavior with Phillips file for an up-to-date version of yt installed from source (updated today) on Ubuntu (stock python 2.7, no pip, no conda):
ds = yt.load("DensTurbMag_hdf5_plt_cnt_0015") yt : [INFO ] 2016-10-19 10:55:08,720 Parameters: current_time = 4.66839074012e-313 yt : [INFO ] 2016-10-19 10:55:08,720 Parameters: domain_dimensions = [1113971554 -402653184 1108822134] yt : [INFO ] 2016-10-19 10:55:08,721 Parameters: domain_left_edge = [ -2.00000000e+18 -2.00000000e+18 -2.00000000e+18] yt : [INFO ] 2016-10-19 10:55:08,721 Parameters: domain_right_edge = [ 2.00000000e+18 2.00000000e+18 2.00000000e+18] yt : [INFO ] 2016-10-19 10:55:08,721 Parameters: cosmological_simulation = 0.0
Best regards, Dominik 2016-10-18 17:48 GMT+02:00 Alex Hill <ashill@haverford.edu>:
Hi John and Phillip,
I’m actively using yt with both Flash 4 and 2.5 data files now with up to date yt (3.3.1) on both my Mac laptop and NASA’s pleiades data analysis nodes, installed via pip; I have no trouble reading them or the test files Phillip uploaded. Maybe try installing with pip instead of conda?
This is the output I get for the _0015 test file:
yt : [INFO ] 2016-10-18 11:43:55,739 Parameters: current_time = 751000000000.0 yt : [INFO ] 2016-10-18 11:43:55,739 Parameters: domain_dimensions = [8 8 8] yt : [INFO ] 2016-10-18 11:43:55,740 Parameters: domain_left_edge = [ -2.00000000e+18 -2.00000000e+18 -2.00000000e+18] yt : [INFO ] 2016-10-18 11:43:55,741 Parameters: domain_right_edge = [ 2.00000000e+18 2.00000000e+18 2.00000000e+18] yt : [INFO ] 2016-10-18 11:43:55,742 Parameters: cosmological_simulation = 0.0
Happy to help diagnose further.
-Alex
--------- Alex Hill Postdoctoral Research Associate Office: KINSC Link L106 Department of Astronomy, Haverford College 370 Lancaster Ave, Haverford, PA 19041 USA phone: +1 484 297 2136 email/iMessage: ashill@haverford.edu
On Oct 18, 2016, at 10:12, John ZuHone <jzuhone@gmail.com> wrote:
Hi Nathan and Phillip,
Since I am nominally the FLASH guy and used to run 2.5 I can check this one out.
================================ John ZuHone, Chandra/ACIS Operations Harvard-Smithsonian Center for Astrophysics
60 Garden St., MS-67 (w) 617-496-1816 Cambridge, MA 02138. (m) 781-708-5004
john.zuhone@cfa.harvard.edu http://hea-www.cfa.harvard.edu/~jzuhone ================================
On Oct 18, 2016, at 10:11 AM, Philipp <keule-phg@gmx.de> wrote:
Hi Nathan,
thank you for the fast reply. Sorry, I didn’t check the sample data for F2.5 files. I uploaded a small test file here: http://use.yt/upload/b895e885 which is at time 0 (maybe not a very clever idea if the time is read incorrectly) and another file ( http://use.yt/upload/1106d6c4) at time 23.8 kyr. The block size is [8,8,8].
With best regards, Philipp
On 18 Oct 2016, at 15:37, Nathan Goldbaum <nathan12343@gmail.com> wrote:
Hi Phillip,
I don't think we have any Flash2.5 test data on yt-project.org/data. We use the datasets there in our test suite, so if we don't have any such data that means that our support for a data format isn't tested and is subject to breakage.
Would you be willing to share some test data that the yt project can use for debugging purposes? It should be relatively small, should trigger the error you're seeing, and should be ok to host publicly on yt-project.org/data and use in our test suite. If so, we can look into restoring support and making sure we don't regress again.
You can share the data using e.g. Dropbox, google drive, or by using the yt curldrop:
https://docs.hub.yt/services.html#curldrop
Nathan
On Tuesday, October 18, 2016, Philipp Girichidis <keule-phg@gmx.de> wrote: Dear YT users,
I upgraded my version of YT from an older version that still installed the libraries on its own (version downloaded 2016/06/21) to a more recent version using yt-conda. With the new version I cannot read old FLASH2.5 data any more. The previous version of yt gave me the correct output
yt : [INFO ] 2016-09-09 16:37:59,285 Parameters: current_time = 1260000000.0 yt : [INFO ] 2016-09-09 16:37:59,286 Parameters: domain_dimensions = [8 8 8] yt : [INFO ] 2016-09-09 16:37:59,289 Parameters: domain_left_edge = [ -4.00000000e+17 -4.00000000e+17 -4.00000000e+17] yt : [INFO ] 2016-09-09 16:37:59,290 Parameters: domain_right_edge = [ 4.00000000e+17 4.00000000e+17 4.00000000e+17] yt : [INFO ] 2016-09-09 16:37:59,291 Parameters: cosmological_simulation = 0.0
the new version does not read some of the numbers properly,
yt : [INFO ] 2016-09-09 16:39:11,692 Parameters: current_time = 1.06099789548e-313 yt : [INFO ] 2016-09-09 16:39:11,692 Parameters: domain_dimensions = [1104332420 0 1102957925] yt : [INFO ] 2016-09-09 16:39:11,693 Parameters: domain_left_edge = [ -4.00000000e+17 -4.00000000e+17 -4.00000000e+17] yt : [INFO ] 2016-09-09 16:39:11,694 Parameters: domain_right_edge = [ 4.00000000e+17 4.00000000e+17 4.00000000e+17] yt : [INFO ] 2016-09-09 16:39:11,695 Parameters: cosmological_simulation = 0.0
The time value as well as the dimensions are messed up and thus yt crashes with a memory error at some point. To me it looks like a conversion problem between big and little endian or some similar conversion problem between single and double precision. I tried various combinations of MAC and linux and the problem does not seem to operating system dependent.
Quite a few functions have changed in terms of reading particle data, but the simulation does not contain particles. I tried to find the part of the code that reads the domain_dimensions and looked at file yt-conda/lib/python2.7/site-packages/yt/frontends/flash/ data_structures.py where in line 99ff the block dimensions are read. But this part of the code is not executed (checked with print statements in iyt). Loading the hdf5 file with h5py and using the code in line 104 gives me the correct numbers, but I guess yt reads the data from somewhere else?
Any hints on how the hierarchy of reading data is organized or where to search? I know FLASH2.5 is very old but we still have a lot of (quite expensive) F2.5 data.
With best regards, Philipp Girichidis _______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org _______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
-- Dominik Derigs I. Physikalisches Institut Universität zu Köln Zülpicher Straße 77 50937 Köln GERMANY https://hera.ph1.uni-koeln.de/~derigs/ Tel. (+49|0) 221 470-8352 Fax. (+49|0) 221 470-5162 Diese Email ist vertraulich und nur für den angegebenen Empfänger bestimmt. Zugang, Freigabe, die Kopie, die Verteilung oder Weiterleitung durch jemand anderen außer dem Empfänger selbst ist verboten und kann eine kriminelle Handlung sein. Bitte löschen Sie die Email, wenn Sie sie durch einen Fehler erhalten haben und informieren Sie den Absender. This email and any files transmitted with it may contain confidential and/or privileged material and is intended only for the person or entity to which it is addressed. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited. If you have received this email in error, please notify the sender immediately and delete this material from all known records.
On 10/18/2016 09:11 AM, Philipp wrote:
Hi Nathan,
thank you for the fast reply. Sorry, I didn’t check the sample data for F2.5 files. I uploaded a small test file here: http://use.yt/upload/b895e885 <http://use.yt/upload/b895e885> which is at time 0 (maybe not a very clever idea if the time is read incorrectly) and another file (http://use.yt/upload/1106d6c4 <http://use.yt/upload/1106d6c4>) at time 23.8 kyr. The block size is [8,8,8].
With best regards, Philipp
Hi Phillip, can I add those datasets to yt's testing suite? That mandates having them publicly available at http://yt-project.org/data/ I've also opened PR 2421 [1] that hopefully will fix the issue you're seeing. Cheers, Kacper [1] https://bitbucket.org/yt_analysis/yt/pull-requests/2421
On 18 Oct 2016, at 15:37, Nathan Goldbaum <nathan12343@gmail.com> wrote:
Hi Phillip,
I don't think we have any Flash2.5 test data on yt-project.org/data <http://yt-project.org/data>. We use the datasets there in our test suite, so if we don't have any such data that means that our support for a data format isn't tested and is subject to breakage.
Would you be willing to share some test data that the yt project can use for debugging purposes? It should be relatively small, should trigger the error you're seeing, and should be ok to host publicly on yt-project.org/data <http://yt-project.org/data> and use in our test suite. If so, we can look into restoring support and making sure we don't regress again.
You can share the data using e.g. Dropbox, google drive, or by using the yt curldrop:
https://docs.hub.yt/services.html#curldrop <https://docs.hub.yt/services.html#curldrop>
Nathan
On Tuesday, October 18, 2016, Philipp Girichidis <keule-phg@gmx.de <mailto:keule-phg@gmx.de>> wrote: Dear YT users,
I upgraded my version of YT from an older version that still installed the libraries on its own (version downloaded 2016/06/21) to a more recent version using yt-conda. With the new version I cannot read old FLASH2.5 data any more. The previous version of yt gave me the correct output
yt : [INFO ] 2016-09-09 16:37:59,285 Parameters: current_time = 1260000000.0 yt : [INFO ] 2016-09-09 16:37:59,286 Parameters: domain_dimensions = [8 8 8] yt : [INFO ] 2016-09-09 16:37:59,289 Parameters: domain_left_edge = [ -4.00000000e+17 -4.00000000e+17 -4.00000000e+17] yt : [INFO ] 2016-09-09 16:37:59,290 Parameters: domain_right_edge = [ 4.00000000e+17 4.00000000e+17 4.00000000e+17] yt : [INFO ] 2016-09-09 16:37:59,291 Parameters: cosmological_simulation = 0.0
the new version does not read some of the numbers properly,
yt : [INFO ] 2016-09-09 16:39:11,692 Parameters: current_time = 1.06099789548e-313 yt : [INFO ] 2016-09-09 16:39:11,692 Parameters: domain_dimensions = [1104332420 0 1102957925] yt : [INFO ] 2016-09-09 16:39:11,693 Parameters: domain_left_edge = [ -4.00000000e+17 -4.00000000e+17 -4.00000000e+17] yt : [INFO ] 2016-09-09 16:39:11,694 Parameters: domain_right_edge = [ 4.00000000e+17 4.00000000e+17 4.00000000e+17] yt : [INFO ] 2016-09-09 16:39:11,695 Parameters: cosmological_simulation = 0.0
The time value as well as the dimensions are messed up and thus yt crashes with a memory error at some point. To me it looks like a conversion problem between big and little endian or some similar conversion problem between single and double precision. I tried various combinations of MAC and linux and the problem does not seem to operating system dependent.
Quite a few functions have changed in terms of reading particle data, but the simulation does not contain particles. I tried to find the part of the code that reads the domain_dimensions and looked at file yt-conda/lib/python2.7/site-packages/yt/frontends/flash/data_structures.py
where in line 99ff the block dimensions are read. But this part of the code is not executed (checked with print statements in iyt). Loading the hdf5 file with h5py and using the code in line 104 gives me the correct numbers, but I guess yt reads the data from somewhere else?
Any hints on how the hierarchy of reading data is organized or where to search? I know FLASH2.5 is very old but we still have a lot of (quite expensive) F2.5 data.
With best regards, Philipp Girichidis _______________________________________________ yt-users mailing list yt-users@lists.spacepope.org <javascript:;> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org <http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org> _______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
Dear Kacper, sure you can add those files to the official testing suite! Dear Alex, I installed yt on a “clean” machine using pip and I still get the error that also Dominik confirmed. Thank you and best regards, Philipp
On 19 Oct 2016, at 17:33, Kacper Kowalik <xarthisius.kk@gmail.com> wrote:
On 10/18/2016 09:11 AM, Philipp wrote:
Hi Nathan,
thank you for the fast reply. Sorry, I didn’t check the sample data for F2.5 files. I uploaded a small test file here: http://use.yt/upload/b895e885 <http://use.yt/upload/b895e885> which is at time 0 (maybe not a very clever idea if the time is read incorrectly) and another file (http://use.yt/upload/1106d6c4 <http://use.yt/upload/1106d6c4>) at time 23.8 kyr. The block size is [8,8,8].
With best regards, Philipp
Hi Phillip, can I add those datasets to yt's testing suite? That mandates having them publicly available at http://yt-project.org/data/
I've also opened PR 2421 [1] that hopefully will fix the issue you're seeing.
Cheers, Kacper
[1] https://bitbucket.org/yt_analysis/yt/pull-requests/2421
On 18 Oct 2016, at 15:37, Nathan Goldbaum <nathan12343@gmail.com> wrote:
Hi Phillip,
I don't think we have any Flash2.5 test data on yt-project.org/data <http://yt-project.org/data>. We use the datasets there in our test suite, so if we don't have any such data that means that our support for a data format isn't tested and is subject to breakage.
Would you be willing to share some test data that the yt project can use for debugging purposes? It should be relatively small, should trigger the error you're seeing, and should be ok to host publicly on yt-project.org/data <http://yt-project.org/data> and use in our test suite. If so, we can look into restoring support and making sure we don't regress again.
You can share the data using e.g. Dropbox, google drive, or by using the yt curldrop:
https://docs.hub.yt/services.html#curldrop <https://docs.hub.yt/services.html#curldrop>
Nathan
On Tuesday, October 18, 2016, Philipp Girichidis <keule-phg@gmx.de <mailto:keule-phg@gmx.de>> wrote: Dear YT users,
I upgraded my version of YT from an older version that still installed the libraries on its own (version downloaded 2016/06/21) to a more recent version using yt-conda. With the new version I cannot read old FLASH2.5 data any more. The previous version of yt gave me the correct output
yt : [INFO ] 2016-09-09 16:37:59,285 Parameters: current_time = 1260000000.0 yt : [INFO ] 2016-09-09 16:37:59,286 Parameters: domain_dimensions = [8 8 8] yt : [INFO ] 2016-09-09 16:37:59,289 Parameters: domain_left_edge = [ -4.00000000e+17 -4.00000000e+17 -4.00000000e+17] yt : [INFO ] 2016-09-09 16:37:59,290 Parameters: domain_right_edge = [ 4.00000000e+17 4.00000000e+17 4.00000000e+17] yt : [INFO ] 2016-09-09 16:37:59,291 Parameters: cosmological_simulation = 0.0
the new version does not read some of the numbers properly,
yt : [INFO ] 2016-09-09 16:39:11,692 Parameters: current_time = 1.06099789548e-313 yt : [INFO ] 2016-09-09 16:39:11,692 Parameters: domain_dimensions = [1104332420 0 1102957925] yt : [INFO ] 2016-09-09 16:39:11,693 Parameters: domain_left_edge = [ -4.00000000e+17 -4.00000000e+17 -4.00000000e+17] yt : [INFO ] 2016-09-09 16:39:11,694 Parameters: domain_right_edge = [ 4.00000000e+17 4.00000000e+17 4.00000000e+17] yt : [INFO ] 2016-09-09 16:39:11,695 Parameters: cosmological_simulation = 0.0
The time value as well as the dimensions are messed up and thus yt crashes with a memory error at some point. To me it looks like a conversion problem between big and little endian or some similar conversion problem between single and double precision. I tried various combinations of MAC and linux and the problem does not seem to operating system dependent.
Quite a few functions have changed in terms of reading particle data, but the simulation does not contain particles. I tried to find the part of the code that reads the domain_dimensions and looked at file yt-conda/lib/python2.7/site-packages/yt/frontends/flash/data_structures.py
where in line 99ff the block dimensions are read. But this part of the code is not executed (checked with print statements in iyt). Loading the hdf5 file with h5py and using the code in line 104 gives me the correct numbers, but I guess yt reads the data from somewhere else?
Any hints on how the hierarchy of reading data is organized or where to search? I know FLASH2.5 is very old but we still have a lot of (quite expensive) F2.5 data.
With best regards, Philipp Girichidis _______________________________________________ yt-users mailing list yt-users@lists.spacepope.org <javascript:;> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org <http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org> _______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
So it turns out this is due to a regression in the h5py library: https://github.com/h5py/h5py/issues/764 If you'd like a quick workaround without Kacper's pull request, you just need to forcibly downgrade h5py: $ conda uninstall h5py $ conda install h5py==2.5.0 or if you use pip: $ pip uninstall h5py $ pip install h5py==2.5.0 Hopefully Kacper's fix will be in by the time we release yt 3.3.2 and you'll be able to upgrade h5py once again and hopefully the h5py bug will be fixed by the time the next version fo h5py comes out. -Nathan On Wed, Oct 19, 2016 at 11:05 AM, Philipp <keule-phg@gmx.de> wrote:
Dear Kacper,
sure you can add those files to the official testing suite!
Dear Alex,
I installed yt on a “clean” machine using pip and I still get the error that also Dominik confirmed.
Thank you and best regards, Philipp
On 19 Oct 2016, at 17:33, Kacper Kowalik <xarthisius.kk@gmail.com> wrote:
On 10/18/2016 09:11 AM, Philipp wrote:
Hi Nathan,
thank you for the fast reply. Sorry, I didn’t check the sample data for F2.5 files. I uploaded a small test file here: http://use.yt/upload/b895e885 <http://use.yt/upload/b895e885> which is at time 0 (maybe not a very clever idea if the time is read incorrectly) and another file (http://use.yt/upload/1106d6c4 <http://use.yt/upload/1106d6c4>) at time 23.8 kyr. The block size is [8,8,8].
With best regards, Philipp
Hi Phillip, can I add those datasets to yt's testing suite? That mandates having them publicly available at http://yt-project.org/data/
I've also opened PR 2421 [1] that hopefully will fix the issue you're seeing.
Cheers, Kacper
[1] https://bitbucket.org/yt_analysis/yt/pull-requests/2421
On 18 Oct 2016, at 15:37, Nathan Goldbaum <nathan12343@gmail.com> wrote:
Hi Phillip,
I don't think we have any Flash2.5 test data on yt-project.org/data <http://yt-project.org/data>. We use the datasets there in our test suite, so if we don't have any such data that means that our support for a data format isn't tested and is subject to breakage.
Would you be willing to share some test data that the yt project can use for debugging purposes? It should be relatively small, should trigger the error you're seeing, and should be ok to host publicly on yt-project.org/data <http://yt-project.org/data> and use in our test suite. If so, we can look into restoring support and making sure we don't regress again.
You can share the data using e.g. Dropbox, google drive, or by using the yt curldrop:
https://docs.hub.yt/services.html#curldrop <https://docs.hub.yt/services.html#curldrop>
Nathan
On Tuesday, October 18, 2016, Philipp Girichidis <keule-phg@gmx.de <mailto:keule-phg@gmx.de>> wrote: Dear YT users,
I upgraded my version of YT from an older version that still installed the libraries on its own (version downloaded 2016/06/21) to a more recent version using yt-conda. With the new version I cannot read old FLASH2.5 data any more. The previous version of yt gave me the correct output
yt : [INFO ] 2016-09-09 16:37:59,285 Parameters: current_time = 1260000000.0 yt : [INFO ] 2016-09-09 16:37:59,286 Parameters: domain_dimensions = [8 8 8] yt : [INFO ] 2016-09-09 16:37:59,289 Parameters: domain_left_edge = [ -4.00000000e+17 -4.00000000e+17 -4.00000000e+17] yt : [INFO ] 2016-09-09 16:37:59,290 Parameters: domain_right_edge = [ 4.00000000e+17 4.00000000e+17 4.00000000e+17] yt : [INFO ] 2016-09-09 16:37:59,291 Parameters: cosmological_simulation = 0.0
the new version does not read some of the numbers properly,
yt : [INFO ] 2016-09-09 16:39:11,692 Parameters: current_time = 1.06099789548e-313 yt : [INFO ] 2016-09-09 16:39:11,692 Parameters: domain_dimensions = [1104332420 0 1102957925] yt : [INFO ] 2016-09-09 16:39:11,693 Parameters: domain_left_edge = [ -4.00000000e+17 -4.00000000e+17 -4.00000000e+17] yt : [INFO ] 2016-09-09 16:39:11,694 Parameters: domain_right_edge = [ 4.00000000e+17 4.00000000e+17 4.00000000e+17] yt : [INFO ] 2016-09-09 16:39:11,695 Parameters: cosmological_simulation = 0.0
The time value as well as the dimensions are messed up and thus yt crashes with a memory error at some point. To me it looks like a conversion problem between big and little endian or some similar conversion problem between single and double precision. I tried various combinations of MAC and linux and the problem does not seem to operating system dependent.
Quite a few functions have changed in terms of reading particle data, but the simulation does not contain particles. I tried to find the part of the code that reads the domain_dimensions and looked at file yt-conda/lib/python2.7/site-packages/yt/frontends/flash/ data_structures.py
where in line 99ff the block dimensions are read. But this part of the code is not executed (checked with print statements in iyt). Loading the hdf5 file with h5py and using the code in line 104 gives me the correct numbers, but I guess yt reads the data from somewhere else?
Any hints on how the hierarchy of reading data is organized or where to search? I know FLASH2.5 is very old but we still have a lot of (quite expensive) F2.5 data.
With best regards, Philipp Girichidis _______________________________________________ yt-users mailing list yt-users@lists.spacepope.org <javascript:;> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org <http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org> _______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
Hi again, I can confirm that downgrading h5py from 2.6.0 to 2.5.0 fixed the issue for me.
ds = yt.load("DensTurbMag_hdf5_plt_cnt_0015") yt : [INFO ] 2016-10-19 18:56:40,998 Parameters: current_time = 751000000000.0 yt : [INFO ] 2016-10-19 18:56:40,999 Parameters: domain_dimensions = [8 8 8] yt : [INFO ] 2016-10-19 18:56:40,999 Parameters: domain_left_edge = [ -2.00000000e+18 -2.00000000e+18 -2.00000000e+18] yt : [INFO ] 2016-10-19 18:56:40,999 Parameters: domain_right_edge = [ 2.00000000e+18 2.00000000e+18 2.00000000e+18] yt : [INFO ] 2016-10-19 18:56:40,999 Parameters: cosmological_simulation = 0.0
Best regards, Dominik 2016-10-19 18:11 GMT+02:00 Nathan Goldbaum <nathan12343@gmail.com>:
So it turns out this is due to a regression in the h5py library:
https://github.com/h5py/h5py/issues/764
If you'd like a quick workaround without Kacper's pull request, you just need to forcibly downgrade h5py:
$ conda uninstall h5py $ conda install h5py==2.5.0
or if you use pip:
$ pip uninstall h5py $ pip install h5py==2.5.0
Hopefully Kacper's fix will be in by the time we release yt 3.3.2 and you'll be able to upgrade h5py once again and hopefully the h5py bug will be fixed by the time the next version fo h5py comes out.
-Nathan
On Wed, Oct 19, 2016 at 11:05 AM, Philipp <keule-phg@gmx.de> wrote:
Dear Kacper,
sure you can add those files to the official testing suite!
Dear Alex,
I installed yt on a “clean” machine using pip and I still get the error that also Dominik confirmed.
Thank you and best regards, Philipp
On 19 Oct 2016, at 17:33, Kacper Kowalik <xarthisius.kk@gmail.com> wrote:
On 10/18/2016 09:11 AM, Philipp wrote:
Hi Nathan,
thank you for the fast reply. Sorry, I didn’t check the sample data for F2.5 files. I uploaded a small test file here: http://use.yt/upload/b895e885 <http://use.yt/upload/b895e885> which is at time 0 (maybe not a very clever idea if the time is read incorrectly) and another file (http://use.yt/upload/1106d6c4 <http://use.yt/upload/1106d6c4>) at time 23.8 kyr. The block size is [8,8,8].
With best regards, Philipp
Hi Phillip, can I add those datasets to yt's testing suite? That mandates having them publicly available at http://yt-project.org/data/
I've also opened PR 2421 [1] that hopefully will fix the issue you're seeing.
Cheers, Kacper
[1] https://bitbucket.org/yt_analysis/yt/pull-requests/2421
On 18 Oct 2016, at 15:37, Nathan Goldbaum <nathan12343@gmail.com> wrote:
Hi Phillip,
I don't think we have any Flash2.5 test data on yt-project.org/data <http://yt-project.org/data>. We use the datasets there in our test suite, so if we don't have any such data that means that our support for a data format isn't tested and is subject to breakage.
Would you be willing to share some test data that the yt project can use for debugging purposes? It should be relatively small, should trigger the error you're seeing, and should be ok to host publicly on yt-project.org/data <http://yt-project.org/data> and use in our test suite. If so, we can look into restoring support and making sure we don't regress again.
You can share the data using e.g. Dropbox, google drive, or by using the yt curldrop:
https://docs.hub.yt/services.html#curldrop <https://docs.hub.yt/services.html#curldrop>
Nathan
On Tuesday, October 18, 2016, Philipp Girichidis <keule-phg@gmx.de <mailto:keule-phg@gmx.de>> wrote: Dear YT users,
I upgraded my version of YT from an older version that still installed the libraries on its own (version downloaded 2016/06/21) to a more recent version using yt-conda. With the new version I cannot read old FLASH2.5 data any more. The previous version of yt gave me the correct output
yt : [INFO ] 2016-09-09 16:37:59,285 Parameters: current_time = 1260000000.0 yt : [INFO ] 2016-09-09 16:37:59,286 Parameters: domain_dimensions = [8 8 8] yt : [INFO ] 2016-09-09 16:37:59,289 Parameters: domain_left_edge = [ -4.00000000e+17 -4.00000000e+17 -4.00000000e+17] yt : [INFO ] 2016-09-09 16:37:59,290 Parameters: domain_right_edge = [ 4.00000000e+17 4.00000000e+17 4.00000000e+17] yt : [INFO ] 2016-09-09 16:37:59,291 Parameters: cosmological_simulation = 0.0
the new version does not read some of the numbers properly,
yt : [INFO ] 2016-09-09 16:39:11,692 Parameters: current_time = 1.06099789548e-313 yt : [INFO ] 2016-09-09 16:39:11,692 Parameters: domain_dimensions = [1104332420 0 1102957925] yt : [INFO ] 2016-09-09 16:39:11,693 Parameters: domain_left_edge = [ -4.00000000e+17 -4.00000000e+17 -4.00000000e+17] yt : [INFO ] 2016-09-09 16:39:11,694 Parameters: domain_right_edge = [ 4.00000000e+17 4.00000000e+17 4.00000000e+17] yt : [INFO ] 2016-09-09 16:39:11,695 Parameters: cosmological_simulation = 0.0
The time value as well as the dimensions are messed up and thus yt crashes with a memory error at some point. To me it looks like a conversion problem between big and little endian or some similar conversion problem between single and double precision. I tried various combinations of MAC and linux and the problem does not seem to operating system dependent.
Quite a few functions have changed in terms of reading particle data, but the simulation does not contain particles. I tried to find the part of the code that reads the domain_dimensions and looked at file yt-conda/lib/python2.7/site-packages/yt/frontends/flash/data _structures.py
where in line 99ff the block dimensions are read. But this part of the code is not executed (checked with print statements in iyt). Loading the hdf5 file with h5py and using the code in line 104 gives me the correct numbers, but I guess yt reads the data from somewhere else?
Any hints on how the hierarchy of reading data is organized or where to search? I know FLASH2.5 is very old but we still have a lot of (quite expensive) F2.5 data.
With best regards, Philipp Girichidis _______________________________________________ yt-users mailing list yt-users@lists.spacepope.org <javascript:;> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org <http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org> _______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
-- Dominik Derigs I. Physikalisches Institut Universität zu Köln Zülpicher Straße 77 50937 Köln GERMANY https://hera.ph1.uni-koeln.de/~derigs/ Tel. (+49|0) 221 470-8352 Fax. (+49|0) 221 470-5162 Diese Email ist vertraulich und nur für den angegebenen Empfänger bestimmt. Zugang, Freigabe, die Kopie, die Verteilung oder Weiterleitung durch jemand anderen außer dem Empfänger selbst ist verboten und kann eine kriminelle Handlung sein. Bitte löschen Sie die Email, wenn Sie sie durch einen Fehler erhalten haben und informieren Sie den Absender. This email and any files transmitted with it may contain confidential and/or privileged material and is intended only for the person or entity to which it is addressed. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited. If you have received this email in error, please notify the sender immediately and delete this material from all known records.
Dear all, after some fights with dependencies and pip I finally managed to get a working version of YT using h5py 2.5.0. Thank you very much for the fast support. Philipp
On 19 Oct 2016, at 18:57, Dominik Derigs <derigs@ph1.uni-koeln.de> wrote:
Hi again,
I can confirm that downgrading h5py from 2.6.0 to 2.5.0 fixed the issue for me.
ds = yt.load("DensTurbMag_hdf5_plt_cnt_0015") yt : [INFO ] 2016-10-19 18:56:40,998 Parameters: current_time = 751000000000.0 yt : [INFO ] 2016-10-19 18:56:40,999 Parameters: domain_dimensions = [8 8 8] yt : [INFO ] 2016-10-19 18:56:40,999 Parameters: domain_left_edge = [ -2.00000000e+18 -2.00000000e+18 -2.00000000e+18] yt : [INFO ] 2016-10-19 18:56:40,999 Parameters: domain_right_edge = [ 2.00000000e+18 2.00000000e+18 2.00000000e+18] yt : [INFO ] 2016-10-19 18:56:40,999 Parameters: cosmological_simulation = 0.0
Best regards, Dominik
2016-10-19 18:11 GMT+02:00 Nathan Goldbaum <nathan12343@gmail.com <mailto:nathan12343@gmail.com>>: So it turns out this is due to a regression in the h5py library:
https://github.com/h5py/h5py/issues/764 <https://github.com/h5py/h5py/issues/764>
If you'd like a quick workaround without Kacper's pull request, you just need to forcibly downgrade h5py:
$ conda uninstall h5py $ conda install h5py==2.5.0
or if you use pip:
$ pip uninstall h5py $ pip install h5py==2.5.0
Hopefully Kacper's fix will be in by the time we release yt 3.3.2 and you'll be able to upgrade h5py once again and hopefully the h5py bug will be fixed by the time the next version fo h5py comes out.
-Nathan
On Wed, Oct 19, 2016 at 11:05 AM, Philipp <keule-phg@gmx.de <mailto:keule-phg@gmx.de>> wrote: Dear Kacper,
sure you can add those files to the official testing suite!
Dear Alex,
I installed yt on a “clean” machine using pip and I still get the error that also Dominik confirmed.
Thank you and best regards, Philipp
On 19 Oct 2016, at 17:33, Kacper Kowalik <xarthisius.kk@gmail.com <mailto:xarthisius.kk@gmail.com>> wrote:
On 10/18/2016 09:11 AM, Philipp wrote:
Hi Nathan,
thank you for the fast reply. Sorry, I didn’t check the sample data for F2.5 files. I uploaded a small test file here: http://use.yt/upload/b895e885 <http://use.yt/upload/b895e885> <http://use.yt/upload/b895e885 <http://use.yt/upload/b895e885>> which is at time 0 (maybe not a very clever idea if the time is read incorrectly) and another file (http://use.yt/upload/1106d6c4 <http://use.yt/upload/1106d6c4> <http://use.yt/upload/1106d6c4 <http://use.yt/upload/1106d6c4>>) at time 23.8 kyr. The block size is [8,8,8].
With best regards, Philipp
Hi Phillip, can I add those datasets to yt's testing suite? That mandates having them publicly available at http://yt-project.org/data/ <http://yt-project.org/data/>
I've also opened PR 2421 [1] that hopefully will fix the issue you're seeing.
Cheers, Kacper
[1] https://bitbucket.org/yt_analysis/yt/pull-requests/2421 <https://bitbucket.org/yt_analysis/yt/pull-requests/2421>
On 18 Oct 2016, at 15:37, Nathan Goldbaum <nathan12343@gmail.com <mailto:nathan12343@gmail.com>> wrote:
Hi Phillip,
I don't think we have any Flash2.5 test data on yt-project.org/data <http://yt-project.org/data> <http://yt-project.org/data <http://yt-project.org/data>>. We use the datasets there in our test suite, so if we don't have any such data that means that our support for a data format isn't tested and is subject to breakage.
Would you be willing to share some test data that the yt project can use for debugging purposes? It should be relatively small, should trigger the error you're seeing, and should be ok to host publicly on yt-project.org/data <http://yt-project.org/data> <http://yt-project.org/data <http://yt-project.org/data>> and use in our test suite. If so, we can look into restoring support and making sure we don't regress again.
You can share the data using e.g. Dropbox, google drive, or by using the yt curldrop:
https://docs.hub.yt/services.html#curldrop <https://docs.hub.yt/services.html#curldrop> <https://docs.hub.yt/services.html#curldrop <https://docs.hub.yt/services.html#curldrop>>
Nathan
On Tuesday, October 18, 2016, Philipp Girichidis <keule-phg@gmx.de <mailto:keule-phg@gmx.de> <mailto:keule-phg@gmx.de <mailto:keule-phg@gmx.de>>> wrote: Dear YT users,
I upgraded my version of YT from an older version that still installed the libraries on its own (version downloaded 2016/06/21) to a more recent version using yt-conda. With the new version I cannot read old FLASH2.5 data any more. The previous version of yt gave me the correct output
yt : [INFO ] 2016-09-09 16:37:59,285 Parameters: current_time = 1260000000.0 yt : [INFO ] 2016-09-09 16:37:59,286 Parameters: domain_dimensions = [8 8 8] yt : [INFO ] 2016-09-09 16:37:59,289 Parameters: domain_left_edge = [ -4.00000000e+17 -4.00000000e+17 -4.00000000e+17] yt : [INFO ] 2016-09-09 16:37:59,290 Parameters: domain_right_edge = [ 4.00000000e+17 4.00000000e+17 4.00000000e+17] yt : [INFO ] 2016-09-09 16:37:59,291 Parameters: cosmological_simulation = 0.0
the new version does not read some of the numbers properly,
yt : [INFO ] 2016-09-09 16:39:11,692 Parameters: current_time = 1.06099789548e-313 yt : [INFO ] 2016-09-09 16:39:11,692 Parameters: domain_dimensions = [1104332420 0 1102957925] yt : [INFO ] 2016-09-09 16:39:11,693 Parameters: domain_left_edge = [ -4.00000000e+17 -4.00000000e+17 -4.00000000e+17] yt : [INFO ] 2016-09-09 16:39:11,694 Parameters: domain_right_edge = [ 4.00000000e+17 4.00000000e+17 4.00000000e+17] yt : [INFO ] 2016-09-09 16:39:11,695 Parameters: cosmological_simulation = 0.0
The time value as well as the dimensions are messed up and thus yt crashes with a memory error at some point. To me it looks like a conversion problem between big and little endian or some similar conversion problem between single and double precision. I tried various combinations of MAC and linux and the problem does not seem to operating system dependent.
Quite a few functions have changed in terms of reading particle data, but the simulation does not contain particles. I tried to find the part of the code that reads the domain_dimensions and looked at file yt-conda/lib/python2.7/site-packages/yt/frontends/flash/data_structures.py
where in line 99ff the block dimensions are read. But this part of the code is not executed (checked with print statements in iyt). Loading the hdf5 file with h5py and using the code in line 104 gives me the correct numbers, but I guess yt reads the data from somewhere else?
Any hints on how the hierarchy of reading data is organized or where to search? I know FLASH2.5 is very old but we still have a lot of (quite expensive) F2.5 data.
With best regards, Philipp Girichidis _______________________________________________ yt-users mailing list yt-users@lists.spacepope.org <mailto:yt-users@lists.spacepope.org> <javascript:;> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org <http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org> <http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org <http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org>> _______________________________________________ yt-users mailing list yt-users@lists.spacepope.org <mailto:yt-users@lists.spacepope.org> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org <http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org>
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org <mailto:yt-users@lists.spacepope.org> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org <http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org>
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org <mailto:yt-users@lists.spacepope.org> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org <http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org>
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org <mailto:yt-users@lists.spacepope.org> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org <http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org>
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org <mailto:yt-users@lists.spacepope.org> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org <http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org>
-- Dominik Derigs I. Physikalisches Institut Universität zu Köln Zülpicher Straße 77 50937 Köln GERMANY
https://hera.ph1.uni-koeln.de/~derigs/ <https://hera.ph1.uni-koeln.de/~derigs/>
Tel. (+49|0) 221 470-8352 Fax. (+49|0) 221 470-5162
Diese Email ist vertraulich und nur für den angegebenen Empfänger bestimmt. Zugang, Freigabe, die Kopie, die Verteilung oder Weiterleitung durch jemand anderen außer dem Empfänger selbst ist verboten und kann eine kriminelle Handlung sein. Bitte löschen Sie die Email, wenn Sie sie durch einen Fehler erhalten haben und informieren Sie den Absender.
This email and any files transmitted with it may contain confidential and/or privileged material and is intended only for the person or entity to which it is addressed. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited. If you have received this email in error, please notify the sender immediately and delete this material from all known records. _______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
participants (7)
-
Alex Hill -
Dominik Derigs -
John ZuHone -
Kacper Kowalik -
Nathan Goldbaum -
Philipp -
Philipp Girichidis