Hello All, I am the one who was asking all of the questions about he ART frontend in the yt-users chat a few weeks ago. I have been probing around the front end files and believe I fixed the file loading problem I outlined here. https://mail.python.org/mm3/archives/list/yt-users@python.org/thread/RV5HY2O... I have a forked version here. https://github.com/sflarkin/yt There are two changes. One is a simple change of a / to a // to fix a num_pages call. https://github.com/yt-project/yt/commit/f233658b153e0ee7b31b31f121488a4257cf... The second is the import one. It appears that in the _find_files function of the art data structures folder had two searches for matching files. The first one works as intended, finding the files with the correct number and data extension. The second just looks for the same data extension, and appears it was just finding all of the files, and then returning the last one found, which from my test was always the file number 170 for the PMcrs files. By removing this check, all appears to work, as shown here on this run log. P001 yt : [INFO ] 2018-08-06 14:44:40,241 discovered particle_header:/nobackupp2/sflarkin/VELA07test/PMcrda0.051.DAT P001 yt : [INFO ] 2018-08-06 14:44:40,241 discovered particle_data:/nobackupp2/sflarkin/VELA07test/PMcrs0a0.051.DAT P001 yt : [INFO ] 2018-08-06 14:44:40,242 discovered particle_stars:/nobackupp2/sflarkin/VELA07test/stars_a0.051.dat P001 yt : [INFO ] 2018-08-06 14:44:40,263 Using root level of 14 P001 yt : [INFO ] 2018-08-06 14:44:40,265 Discovered 7 species of particles P001 yt : [INFO ] 2018-08-06 14:44:40,265 Particle populations: 34930688 4943872 855040 139392 21776 2088309 2 P001 yt : [INFO ] 2018-08-06 14:44:40,483 Max level is 08 P001 yt : [INFO ] 2018-08-06 14:44:40,495 Parameters: current_time = 0.2068722930617056 Gyr P001 yt : [INFO ] 2018-08-06 14:44:40,496 Parameters: domain_dimensions = [128 128 128] P001 yt : [INFO ] 2018-08-06 14:44:40,496 Parameters: domain_left_edge = [0. 0. 0.] P001 yt : [INFO ] 2018-08-06 14:44:40,496 Parameters: domain_right_edge = [1. 1. 1.] P001 yt : [INFO ] 2018-08-06 14:44:40,496 Parameters: cosmological_simulation = True P001 yt : [INFO ] 2018-08-06 14:44:40,496 Parameters: current_redshift = 18.578137813635497 P001 yt : [INFO ] 2018-08-06 14:44:40,496 Parameters: omega_lambda = 0.7300000190734863 P001 yt : [INFO ] 2018-08-06 14:44:40,496 Parameters: omega_matter = 0.27000001072883606 P001 yt : [INFO ] 2018-08-06 14:44:40,496 Parameters: hubble_constant = 0.699999988079071 [ 141s] Sending projection requests... [ 143s] Transferring particles to writers... [ 146s] Analyzing for FoF groups... [ 148s] Transferring boundary particles between writers... [ 148s] Linking boundary particles... [ 149s] Analyzing for halos / subhalos... [ 150s] Loading merger tree information... [ 151s] Constructing merger tree... [ 152s] [Success] Done with snapshot 3. [ 152s] Reading 1 blocks for snapshot 4... P001 yt : [INFO ] 2018-08-06 14:45:21,705 discovered particle_header:/nobackupp2/sflarkin/VELA07test/PMcrda0.060.DAT P001 yt : [INFO ] 2018-08-06 14:45:21,705 discovered particle_data:/nobackupp2/sflarkin/VELA07test/PMcrs0a0.060.DAT P001 yt : [INFO ] 2018-08-06 14:45:21,705 discovered particle_stars:/nobackupp2/sflarkin/VELA07test/stars_a0.060.dat P001 yt : [INFO ] 2018-08-06 14:45:21,728 Using root level of 14 P001 yt : [INFO ] 2018-08-06 14:45:21,755 Discovered 7 species of particles P001 yt : [INFO ] 2018-08-06 14:45:21,755 Particle populations: 34930688 4943872 855040 139392 21776 2088309 149 P001 yt : [INFO ] 2018-08-06 14:45:21,948 Max level is 09 P001 yt : [INFO ] 2018-08-06 14:45:21,961 Parameters: current_time = 0.2648085269122017 Gyr P001 yt : [INFO ] 2018-08-06 14:45:21,961 Parameters: domain_dimensions = [128 128 128] P001 yt : [INFO ] 2018-08-06 14:45:21,961 Parameters: domain_left_edge = [0. 0. 0.] P001 yt : [INFO ] 2018-08-06 14:45:21,961 Parameters: domain_right_edge = [1. 1. 1.] P001 yt : [INFO ] 2018-08-06 14:45:21,961 Parameters: cosmological_simulation = True P001 yt : [INFO ] 2018-08-06 14:45:21,961 Parameters: current_redshift = 15.606548426007368 P001 yt : [INFO ] 2018-08-06 14:45:21,961 Parameters: omega_lambda = 0.7300000190734863 P001 yt : [INFO ] 2018-08-06 14:45:21,962 Parameters: omega_matter = 0.27000001072883606 P001 yt : [INFO ] 2018-08-06 14:45:21,962 Parameters: hubble_constant = 0.699999988079071 This shows two snapshots being loaded in parallel, both finding all of the correct files for the stars and PM files. While this appears to fix the problem, I am not sure if that catch is there for some edge case I cannot think of. Is there one?
Hi Sean, Can you propose a pull request so we can continue this discussion in that context? It's much easier to talk about code changes in the context of a pull request. I will ping Kenza Arraki about this, who is the current contact for the ART frontend. -Nathan On Wed, Aug 8, 2018 at 2:28 PM, <sflarkin@ucsc.edu> wrote:
Hello All,
I am the one who was asking all of the questions about he ART frontend in the yt-users chat a few weeks ago. I have been probing around the front end files and believe I fixed the file loading problem I outlined here.
https://mail.python.org/mm3/archives/list/yt-users@python.org/thread/ RV5HY2OUEDOBP64H75GY2Z6VDDJUK62Z/
I have a forked version here.
https://github.com/sflarkin/yt
There are two changes. One is a simple change of a / to a // to fix a num_pages call.
https://github.com/yt-project/yt/commit/f233658b153e0ee7b31b31f121488a 4257cf13c7
The second is the import one. It appears that in the _find_files function of the art data structures folder had two searches for matching files. The first one works as intended, finding the files with the correct number and data extension. The second just looks for the same data extension, and appears it was just finding all of the files, and then returning the last one found, which from my test was always the file number 170 for the PMcrs files. By removing this check, all appears to work, as shown here on this run log.
P001 yt : [INFO ] 2018-08-06 14:44:40,241 discovered particle_header:/nobackupp2/sflarkin/VELA07test/PMcrda0.051.DAT P001 yt : [INFO ] 2018-08-06 14:44:40,241 discovered particle_data:/nobackupp2/sflarkin/VELA07test/PMcrs0a0.051.DAT P001 yt : [INFO ] 2018-08-06 14:44:40,242 discovered particle_stars:/nobackupp2/sflarkin/VELA07test/stars_a0.051.dat P001 yt : [INFO ] 2018-08-06 14:44:40,263 Using root level of 14 P001 yt : [INFO ] 2018-08-06 14:44:40,265 Discovered 7 species of particles P001 yt : [INFO ] 2018-08-06 14:44:40,265 Particle populations: 34930688 4943872 855040 139392 21776 2088309 2 P001 yt : [INFO ] 2018-08-06 14:44:40,483 Max level is 08 P001 yt : [INFO ] 2018-08-06 14:44:40,495 Parameters: current_time = 0.2068722930617056 Gyr P001 yt : [INFO ] 2018-08-06 14:44:40,496 Parameters: domain_dimensions = [128 128 128] P001 yt : [INFO ] 2018-08-06 14:44:40,496 Parameters: domain_left_edge = [0. 0. 0.] P001 yt : [INFO ] 2018-08-06 14:44:40,496 Parameters: domain_right_edge = [1. 1. 1.] P001 yt : [INFO ] 2018-08-06 14:44:40,496 Parameters: cosmological_simulation = True P001 yt : [INFO ] 2018-08-06 14:44:40,496 Parameters: current_redshift = 18.578137813635497 P001 yt : [INFO ] 2018-08-06 14:44:40,496 Parameters: omega_lambda = 0.7300000190734863 P001 yt : [INFO ] 2018-08-06 14:44:40,496 Parameters: omega_matter = 0.27000001072883606 P001 yt : [INFO ] 2018-08-06 14:44:40,496 Parameters: hubble_constant = 0.699999988079071 [ 141s] Sending projection requests... [ 143s] Transferring particles to writers... [ 146s] Analyzing for FoF groups... [ 148s] Transferring boundary particles between writers... [ 148s] Linking boundary particles... [ 149s] Analyzing for halos / subhalos... [ 150s] Loading merger tree information... [ 151s] Constructing merger tree... [ 152s] [Success] Done with snapshot 3. [ 152s] Reading 1 blocks for snapshot 4... P001 yt : [INFO ] 2018-08-06 14:45:21,705 discovered particle_header:/nobackupp2/sflarkin/VELA07test/PMcrda0.060.DAT P001 yt : [INFO ] 2018-08-06 14:45:21,705 discovered particle_data:/nobackupp2/sflarkin/VELA07test/PMcrs0a0.060.DAT P001 yt : [INFO ] 2018-08-06 14:45:21,705 discovered particle_stars:/nobackupp2/sflarkin/VELA07test/stars_a0.060.dat P001 yt : [INFO ] 2018-08-06 14:45:21,728 Using root level of 14 P001 yt : [INFO ] 2018-08-06 14:45:21,755 Discovered 7 species of particles P001 yt : [INFO ] 2018-08-06 14:45:21,755 Particle populations: 34930688 4943872 855040 139392 21776 2088309 149 P001 yt : [INFO ] 2018-08-06 14:45:21,948 Max level is 09 P001 yt : [INFO ] 2018-08-06 14:45:21,961 Parameters: current_time = 0.2648085269122017 Gyr P001 yt : [INFO ] 2018-08-06 14:45:21,961 Parameters: domain_dimensions = [128 128 128] P001 yt : [INFO ] 2018-08-06 14:45:21,961 Parameters: domain_left_edge = [0. 0. 0.] P001 yt : [INFO ] 2018-08-06 14:45:21,961 Parameters: domain_right_edge = [1. 1. 1.] P001 yt : [INFO ] 2018-08-06 14:45:21,961 Parameters: cosmological_simulation = True P001 yt : [INFO ] 2018-08-06 14:45:21,961 Parameters: current_redshift = 15.606548426007368 P001 yt : [INFO ] 2018-08-06 14:45:21,961 Parameters: omega_lambda = 0.7300000190734863 P001 yt : [INFO ] 2018-08-06 14:45:21,962 Parameters: omega_matter = 0.27000001072883606 P001 yt : [INFO ] 2018-08-06 14:45:21,962 Parameters: hubble_constant = 0.699999988079071
This shows two snapshots being loaded in parallel, both finding all of the correct files for the stars and PM files.
While this appears to fix the problem, I am not sure if that catch is there for some edge case I cannot think of. Is there one? _______________________________________________ yt-dev mailing list -- yt-dev@python.org To unsubscribe send an email to yt-dev-leave@python.org
participants (2)
-
Nathan Goldbaum
-
sflarkin@ucsc.edu