Hi All,
I've started to see a strange error in a simple enough parallel computation.
I can reproduce it with the script below but with some other scripts where the code is embedded it runs fine so it may be a race condition.
from yt.mods import *
pf = load("/wrk/regan/H2/Ctrl/DD0035/Data0035") v, c = pf.h.find_max("Density") print v, c
Thanks in advance for any help!
Cheers, John
Error:
Traceback (most recent call last): P001 yt : [INFO ] 2013-12-29 09:31:27,756 Max Value is 3.67832e-16 at 0.5025423020124483 0.4896353334188484 0.4947452098131160 in grid EnzoGrid_1214 at level 18 (3, 3, 2) Traceback (most recent call last): File "ptest.py", line 5, in <module> P003 yt : [INFO ] 2013-12-29 09:31:27,756 Max Value is 3.67832e-16 at 0.5025423020124483 0.4896353334188484 0.4947452098131160 in grid EnzoGrid_1214 at level 18 (3, 3, 2) File "ptest.py", line 5, in <module> Traceback (most recent call last): File "ptest.py", line 5, in <module> P002 yt : [INFO ] 2013-12-29 09:31:27,756 Max Value is 3.67832e-16 at 0.5025423020124483 0.4896353334188484 0.4947452098131160 in grid EnzoGrid_1214 at level 18 (3, 3, 2) Traceback (most recent call last): P004 yt : [INFO ] 2013-12-29 09:31:27,756 Max Value is 3.67832e-16 at 0.5025423020124483 0.4896353334188484 0.4947452098131160 in grid EnzoGrid_1214 at level 18 (3, 3, 2) Traceback (most recent call last): File "ptest.py", line 5, in <module> P005 yt : [INFO ] 2013-12-29 09:31:27,756 Max Value is 3.67832e-16 at 0.5025423020124483 0.4896353334188484 0.4947452098131160 in grid EnzoGrid_1214 at level 18 (3, 3, 2) Traceback (most recent call last): File "ptest.py", line 5, in <module> P006 yt : [INFO ] 2013-12-29 09:31:27,756 Max Value is 3.67832e-16 at 0.5025423020124483 0.4896353334188484 0.4947452098131160 in grid EnzoGrid_1214 at level 18 (3, 3, 2) Traceback (most recent call last): File "ptest.py", line 5, in <module> Traceback (most recent call last): File "ptest.py", line 5, in <module> Traceback (most recent call last): File "ptest.py", line 5, in <module> File "ptest.py", line 5, in <module> v, c = pf.h.find_max("Density") v, c = pf.h.find_max("Density") v, c = pf.h.find_max("Density") v, c = pf.h.find_max("Density") File "/homeappl/home/regan/appl_taito/YT/yt-x86_64/src/yt-hg/yt/data_objects/object_finding_mixin.py", line 51, in find_max v, c = pf.h.find_max("Density") File "/homeappl/home/regan/appl_taito/YT/yt-x86_64/src/yt-hg/yt/data_objects/object_finding_mixin.py", line 51, in find_max File "/homeappl/home/regan/appl_taito/YT/yt-x86_64/src/yt-hg/yt/data_objects/object_finding_mixin.py", line 51, in find_max File "/homeappl/home/regan/appl_taito/YT/yt-x86_64/src/yt-hg/yt/data_objects/object_finding_mixin.py", line 51, in find_max v, c = pf.h.find_max("Density") File "/homeappl/home/regan/appl_taito/YT/yt-x86_64/src/yt-hg/yt/data_objects/object_finding_mixin.py", line 51, in find_max v, c = pf.h.find_max("Density") v, c = pf.h.find_max("Density") v, c = pf.h.find_max("Density") File "/homeappl/home/regan/appl_taito/YT/yt-x86_64/src/yt-hg/yt/data_objects/object_finding_mixin.py", line 51, in find_max File "/homeappl/home/regan/appl_taito/YT/yt-x86_64/src/yt-hg/yt/data_objects/object_finding_mixin.py", line 51, in find_max File "/homeappl/home/regan/appl_taito/YT/yt-x86_64/src/yt-hg/yt/data_objects/object_finding_mixin.py", line 51, in find_max File "/homeappl/home/regan/appl_taito/YT/yt-x86_64/src/yt-hg/yt/data_objects/object_finding_mixin.py", line 51, in find_max mg, mc, mv, pos = self.find_max_cell_location(field, finest_levels) mg, mc, mv, pos = self.find_max_cell_location(field, finest_levels) File "/homeappl/home/regan/appl_taito/YT/yt-x86_64/src/yt-hg/yt/data_objects/object_finding_mixin.py", line 66, in find_max_cell_location mg, mc, mv, pos = self.find_max_cell_location(field, finest_levels) File "/homeappl/home/regan/appl_taito/YT/yt-x86_64/src/yt-hg/yt/data_objects/object_finding_mixin.py", line 66, in find_max_cell_location File "/homeappl/home/regan/appl_taito/YT/yt-x86_64/src/yt-hg/yt/data_objects/object_finding_mixin.py", line 66, in find_max_cell_location mg, mc, mv, pos = self.find_max_cell_location(field, finest_levels)
Hi John,
Can you paste the full traceback using: http://paste.yt-project.org? Also, are you running with mpirun python your_script.py --parallel? The only thought I have at the moment is that there is a mismatch between the MPI you are running mpirun with and the MPI that mpi4py was built with.
Also, which version of yt are you using (the results of "yt instinfo")?
Thanks, Sam
On Sat, Dec 28, 2013 at 11:36 PM, John Regan johnanthonyregan@gmail.comwrote:
Hi All,
I've started to see a strange error in a simple enough parallel computation.
I can reproduce it with the script below but with some other scripts where the code is embedded it runs fine so it may be a race condition.
from yt.mods import *
pf = load("/wrk/regan/H2/Ctrl/DD0035/Data0035") v, c = pf.h.find_max("Density") print v, c
Thanks in advance for any help!
Cheers, John
Error:
Traceback (most recent call last): P001 yt : [INFO ] 2013-12-29 09:31:27,756 Max Value is 3.67832e-16 at 0.5025423020124483 0.4896353334188484 0.4947452098131160 in grid EnzoGrid_1214 at level 18 (3, 3, 2) Traceback (most recent call last): File "ptest.py", line 5, in <module> P003 yt : [INFO ] 2013-12-29 09:31:27,756 Max Value is 3.67832e-16 at 0.5025423020124483 0.4896353334188484 0.4947452098131160 in grid EnzoGrid_1214 at level 18 (3, 3, 2) File "ptest.py", line 5, in <module> Traceback (most recent call last): File "ptest.py", line 5, in <module> P002 yt : [INFO ] 2013-12-29 09:31:27,756 Max Value is 3.67832e-16 at 0.5025423020124483 0.4896353334188484 0.4947452098131160 in grid EnzoGrid_1214 at level 18 (3, 3, 2) Traceback (most recent call last): P004 yt : [INFO ] 2013-12-29 09:31:27,756 Max Value is 3.67832e-16 at 0.5025423020124483 0.4896353334188484 0.4947452098131160 in grid EnzoGrid_1214 at level 18 (3, 3, 2) Traceback (most recent call last): File "ptest.py", line 5, in <module> P005 yt : [INFO ] 2013-12-29 09:31:27,756 Max Value is 3.67832e-16 at 0.5025423020124483 0.4896353334188484 0.4947452098131160 in grid EnzoGrid_1214 at level 18 (3, 3, 2) Traceback (most recent call last): File "ptest.py", line 5, in <module> P006 yt : [INFO ] 2013-12-29 09:31:27,756 Max Value is 3.67832e-16 at 0.5025423020124483 0.4896353334188484 0.4947452098131160 in grid EnzoGrid_1214 at level 18 (3, 3, 2) Traceback (most recent call last): File "ptest.py", line 5, in <module> Traceback (most recent call last): File "ptest.py", line 5, in <module> Traceback (most recent call last): File "ptest.py", line 5, in <module> File "ptest.py", line 5, in <module> v, c = pf.h.find_max("Density") v, c = pf.h.find_max("Density") v, c = pf.h.find_max("Density") v, c = pf.h.find_max("Density") File "/homeappl/home/regan/appl_taito/YT/yt-x86_64/src/yt-hg/yt/data_objects/object_finding_mixin.py", line 51, in find_max v, c = pf.h.find_max("Density") File "/homeappl/home/regan/appl_taito/YT/yt-x86_64/src/yt-hg/yt/data_objects/object_finding_mixin.py", line 51, in find_max File "/homeappl/home/regan/appl_taito/YT/yt-x86_64/src/yt-hg/yt/data_objects/object_finding_mixin.py", line 51, in find_max File "/homeappl/home/regan/appl_taito/YT/yt-x86_64/src/yt-hg/yt/data_objects/object_finding_mixin.py", line 51, in find_max v, c = pf.h.find_max("Density") File "/homeappl/home/regan/appl_taito/YT/yt-x86_64/src/yt-hg/yt/data_objects/object_finding_mixin.py", line 51, in find_max v, c = pf.h.find_max("Density") v, c = pf.h.find_max("Density") v, c = pf.h.find_max("Density") File "/homeappl/home/regan/appl_taito/YT/yt-x86_64/src/yt-hg/yt/data_objects/object_finding_mixin.py", line 51, in find_max File "/homeappl/home/regan/appl_taito/YT/yt-x86_64/src/yt-hg/yt/data_objects/object_finding_mixin.py", line 51, in find_max File "/homeappl/home/regan/appl_taito/YT/yt-x86_64/src/yt-hg/yt/data_objects/object_finding_mixin.py", line 51, in find_max File "/homeappl/home/regan/appl_taito/YT/yt-x86_64/src/yt-hg/yt/data_objects/object_finding_mixin.py", line 51, in find_max mg, mc, mv, pos = self.find_max_cell_location(field, finest_levels) mg, mc, mv, pos = self.find_max_cell_location(field, finest_levels) File "/homeappl/home/regan/appl_taito/YT/yt-x86_64/src/yt-hg/yt/data_objects/object_finding_mixin.py", line 66, in find_max_cell_location mg, mc, mv, pos = self.find_max_cell_location(field, finest_levels) File "/homeappl/home/regan/appl_taito/YT/yt-x86_64/src/yt-hg/yt/data_objects/object_finding_mixin.py", line 66, in find_max_cell_location File "/homeappl/home/regan/appl_taito/YT/yt-x86_64/src/yt-hg/yt/data_objects/object_finding_mixin.py", line 66, in find_max_cell_location mg, mc, mv, pos = self.find_max_cell_location(field, finest_levels)
yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
Hi Sam,
I'm running the script using srun pyyt ptest.py --parallel
That could well be picking up the system mpirun rather than the mpi that YT was built with - I hadn't considered that. I'll check that. The traceback is at the link below.
http://paste.yt-project.org/show/4174/
The version of YT is the dev version - that's what's sourced when the batch script is run. I've tried with the stable version as well and got the same result. I think you might be on to something with the mpirun library issue though.
Thanks for the tip!
John
(yt-x86_64)[regan@taito-login3 Projections]$ yt instinfo
yt module located at: /homeappl/home/regan/appl_taito/YT/yt-x86_64/src/yt-hg The supplemental repositories are located at: /homeappl/home/regan/appl_taito/YT/yt-x86_64/src/yt-supplemental
The current version of the code is:
--- ec9a5117fc08 ---
This installation CAN be automatically updated. yt dependencies were last updated on Tue Dec 3 17:12:02 EET 2013
To update all dependencies, run "yt update --all".
On Sun, Dec 29, 2013 at 8:22 PM, Sam Skillman samskillman@gmail.com wrote:
Hi John,
Can you paste the full traceback using: http://paste.yt-project.org? Also, are you running with mpirun python your_script.py --parallel? The only thought I have at the moment is that there is a mismatch between the MPI you are running mpirun with and the MPI that mpi4py was built with.
Also, which version of yt are you using (the results of "yt instinfo")?
Thanks, Sam
On Sat, Dec 28, 2013 at 11:36 PM, John Regan johnanthonyregan@gmail.comwrote:
Hi All,
I've started to see a strange error in a simple enough parallel computation.
I can reproduce it with the script below but with some other scripts where the code is embedded it runs fine so it may be a race condition.
from yt.mods import *
pf = load("/wrk/regan/H2/Ctrl/DD0035/Data0035") v, c = pf.h.find_max("Density") print v, c
Thanks in advance for any help!
Cheers, John
Error:
Traceback (most recent call last): P001 yt : [INFO ] 2013-12-29 09:31:27,756 Max Value is 3.67832e-16 at 0.5025423020124483 0.4896353334188484 0.4947452098131160 in grid EnzoGrid_1214 at level 18 (3, 3, 2) Traceback (most recent call last): File "ptest.py", line 5, in <module> P003 yt : [INFO ] 2013-12-29 09:31:27,756 Max Value is 3.67832e-16 at 0.5025423020124483 0.4896353334188484 0.4947452098131160 in grid EnzoGrid_1214 at level 18 (3, 3, 2) File "ptest.py", line 5, in <module> Traceback (most recent call last): File "ptest.py", line 5, in <module> P002 yt : [INFO ] 2013-12-29 09:31:27,756 Max Value is 3.67832e-16 at 0.5025423020124483 0.4896353334188484 0.4947452098131160 in grid EnzoGrid_1214 at level 18 (3, 3, 2) Traceback (most recent call last): P004 yt : [INFO ] 2013-12-29 09:31:27,756 Max Value is 3.67832e-16 at 0.5025423020124483 0.4896353334188484 0.4947452098131160 in grid EnzoGrid_1214 at level 18 (3, 3, 2) Traceback (most recent call last): File "ptest.py", line 5, in <module> P005 yt : [INFO ] 2013-12-29 09:31:27,756 Max Value is 3.67832e-16 at 0.5025423020124483 0.4896353334188484 0.4947452098131160 in grid EnzoGrid_1214 at level 18 (3, 3, 2) Traceback (most recent call last): File "ptest.py", line 5, in <module> P006 yt : [INFO ] 2013-12-29 09:31:27,756 Max Value is 3.67832e-16 at 0.5025423020124483 0.4896353334188484 0.4947452098131160 in grid EnzoGrid_1214 at level 18 (3, 3, 2) Traceback (most recent call last): File "ptest.py", line 5, in <module> Traceback (most recent call last): File "ptest.py", line 5, in <module> Traceback (most recent call last): File "ptest.py", line 5, in <module> File "ptest.py", line 5, in <module> v, c = pf.h.find_max("Density") v, c = pf.h.find_max("Density") v, c = pf.h.find_max("Density") v, c = pf.h.find_max("Density") File "/homeappl/home/regan/appl_taito/YT/yt-x86_64/src/yt-hg/yt/data_objects/object_finding_mixin.py", line 51, in find_max v, c = pf.h.find_max("Density") File "/homeappl/home/regan/appl_taito/YT/yt-x86_64/src/yt-hg/yt/data_objects/object_finding_mixin.py", line 51, in find_max File "/homeappl/home/regan/appl_taito/YT/yt-x86_64/src/yt-hg/yt/data_objects/object_finding_mixin.py", line 51, in find_max File "/homeappl/home/regan/appl_taito/YT/yt-x86_64/src/yt-hg/yt/data_objects/object_finding_mixin.py", line 51, in find_max v, c = pf.h.find_max("Density") File "/homeappl/home/regan/appl_taito/YT/yt-x86_64/src/yt-hg/yt/data_objects/object_finding_mixin.py", line 51, in find_max v, c = pf.h.find_max("Density") v, c = pf.h.find_max("Density") v, c = pf.h.find_max("Density") File "/homeappl/home/regan/appl_taito/YT/yt-x86_64/src/yt-hg/yt/data_objects/object_finding_mixin.py", line 51, in find_max File "/homeappl/home/regan/appl_taito/YT/yt-x86_64/src/yt-hg/yt/data_objects/object_finding_mixin.py", line 51, in find_max File "/homeappl/home/regan/appl_taito/YT/yt-x86_64/src/yt-hg/yt/data_objects/object_finding_mixin.py", line 51, in find_max File "/homeappl/home/regan/appl_taito/YT/yt-x86_64/src/yt-hg/yt/data_objects/object_finding_mixin.py", line 51, in find_max mg, mc, mv, pos = self.find_max_cell_location(field, finest_levels) mg, mc, mv, pos = self.find_max_cell_location(field, finest_levels) File "/homeappl/home/regan/appl_taito/YT/yt-x86_64/src/yt-hg/yt/data_objects/object_finding_mixin.py", line 66, in find_max_cell_location mg, mc, mv, pos = self.find_max_cell_location(field, finest_levels) File "/homeappl/home/regan/appl_taito/YT/yt-x86_64/src/yt-hg/yt/data_objects/object_finding_mixin.py", line 66, in find_max_cell_location File "/homeappl/home/regan/appl_taito/YT/yt-x86_64/src/yt-hg/yt/data_objects/object_finding_mixin.py", line 66, in find_max_cell_location mg, mc, mv, pos = self.find_max_cell_location(field, finest_levels)
yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
yt itself does not link against an mpi library. Instead, we use mpi4py to handle bindings to the local mpi installation. To be sure that mpi4py is linked against the correct mpi lbirary, you should first uninstall mpi4py:
$ pip uninstall mpi4py
then download an mpi4py tarball, unzip it, and build it using the correct sysem MPI
$ wget https://mpi4py.googlecode.com/files/mpi4py-1.3.1.tar.gz $ tar xzvf mpi4py-1.3.1.tar.gz $ python setup.py build --mpicc=/where/you/have/mpicc $ python setup.py install
When you buid mpi4py, be sure to specify the path to the MPI compiler for the system MPI package you want to link against. For more detail on building mpi4py, check here: http://mpi4py.scipy.org/docs/usrman/install.html#building
On Sun, Dec 29, 2013 at 11:54 AM, John Regan johnanthonyregan@gmail.comwrote:
Hi Sam,
I'm running the script using srun pyyt ptest.py --parallel
That could well be picking up the system mpirun rather than the mpi that YT was built with - I hadn't considered that. I'll check that. The traceback is at the link below.
http://paste.yt-project.org/show/4174/
The version of YT is the dev version - that's what's sourced when the batch script is run. I've tried with the stable version as well and got the same result. I think you might be on to something with the mpirun library issue though.
Thanks for the tip!
John
(yt-x86_64)[regan@taito-login3 Projections]$ yt instinfo
yt module located at: /homeappl/home/regan/appl_taito/YT/yt-x86_64/src/yt-hg The supplemental repositories are located at: /homeappl/home/regan/appl_taito/YT/yt-x86_64/src/yt-supplemental
The current version of the code is:
ec9a5117fc08
This installation CAN be automatically updated. yt dependencies were last updated on Tue Dec 3 17:12:02 EET 2013
To update all dependencies, run "yt update --all".
On Sun, Dec 29, 2013 at 8:22 PM, Sam Skillman samskillman@gmail.comwrote:
Hi John,
Can you paste the full traceback using: http://paste.yt-project.org? Also, are you running with mpirun python your_script.py --parallel? The only thought I have at the moment is that there is a mismatch between the MPI you are running mpirun with and the MPI that mpi4py was built with.
Also, which version of yt are you using (the results of "yt instinfo")?
Thanks, Sam
On Sat, Dec 28, 2013 at 11:36 PM, John Regan johnanthonyregan@gmail.comwrote:
Hi All,
I've started to see a strange error in a simple enough parallel computation.
I can reproduce it with the script below but with some other scripts where the code is embedded it runs fine so it may be a race condition.
from yt.mods import *
pf = load("/wrk/regan/H2/Ctrl/DD0035/Data0035") v, c = pf.h.find_max("Density") print v, c
Thanks in advance for any help!
Cheers, John
Error:
Traceback (most recent call last): P001 yt : [INFO ] 2013-12-29 09:31:27,756 Max Value is 3.67832e-16 at 0.5025423020124483 0.4896353334188484 0.4947452098131160 in grid EnzoGrid_1214 at level 18 (3, 3, 2) Traceback (most recent call last): File "ptest.py", line 5, in <module> P003 yt : [INFO ] 2013-12-29 09:31:27,756 Max Value is 3.67832e-16 at 0.5025423020124483 0.4896353334188484 0.4947452098131160 in grid EnzoGrid_1214 at level 18 (3, 3, 2) File "ptest.py", line 5, in <module> Traceback (most recent call last): File "ptest.py", line 5, in <module> P002 yt : [INFO ] 2013-12-29 09:31:27,756 Max Value is 3.67832e-16 at 0.5025423020124483 0.4896353334188484 0.4947452098131160 in grid EnzoGrid_1214 at level 18 (3, 3, 2) Traceback (most recent call last): P004 yt : [INFO ] 2013-12-29 09:31:27,756 Max Value is 3.67832e-16 at 0.5025423020124483 0.4896353334188484 0.4947452098131160 in grid EnzoGrid_1214 at level 18 (3, 3, 2) Traceback (most recent call last): File "ptest.py", line 5, in <module> P005 yt : [INFO ] 2013-12-29 09:31:27,756 Max Value is 3.67832e-16 at 0.5025423020124483 0.4896353334188484 0.4947452098131160 in grid EnzoGrid_1214 at level 18 (3, 3, 2) Traceback (most recent call last): File "ptest.py", line 5, in <module> P006 yt : [INFO ] 2013-12-29 09:31:27,756 Max Value is 3.67832e-16 at 0.5025423020124483 0.4896353334188484 0.4947452098131160 in grid EnzoGrid_1214 at level 18 (3, 3, 2) Traceback (most recent call last): File "ptest.py", line 5, in <module> Traceback (most recent call last): File "ptest.py", line 5, in <module> Traceback (most recent call last): File "ptest.py", line 5, in <module> File "ptest.py", line 5, in <module> v, c = pf.h.find_max("Density") v, c = pf.h.find_max("Density") v, c = pf.h.find_max("Density") v, c = pf.h.find_max("Density") File "/homeappl/home/regan/appl_taito/YT/yt-x86_64/src/yt-hg/yt/data_objects/object_finding_mixin.py", line 51, in find_max v, c = pf.h.find_max("Density") File "/homeappl/home/regan/appl_taito/YT/yt-x86_64/src/yt-hg/yt/data_objects/object_finding_mixin.py", line 51, in find_max File "/homeappl/home/regan/appl_taito/YT/yt-x86_64/src/yt-hg/yt/data_objects/object_finding_mixin.py", line 51, in find_max File "/homeappl/home/regan/appl_taito/YT/yt-x86_64/src/yt-hg/yt/data_objects/object_finding_mixin.py", line 51, in find_max v, c = pf.h.find_max("Density") File "/homeappl/home/regan/appl_taito/YT/yt-x86_64/src/yt-hg/yt/data_objects/object_finding_mixin.py", line 51, in find_max v, c = pf.h.find_max("Density") v, c = pf.h.find_max("Density") v, c = pf.h.find_max("Density") File "/homeappl/home/regan/appl_taito/YT/yt-x86_64/src/yt-hg/yt/data_objects/object_finding_mixin.py", line 51, in find_max File "/homeappl/home/regan/appl_taito/YT/yt-x86_64/src/yt-hg/yt/data_objects/object_finding_mixin.py", line 51, in find_max File "/homeappl/home/regan/appl_taito/YT/yt-x86_64/src/yt-hg/yt/data_objects/object_finding_mixin.py", line 51, in find_max File "/homeappl/home/regan/appl_taito/YT/yt-x86_64/src/yt-hg/yt/data_objects/object_finding_mixin.py", line 51, in find_max mg, mc, mv, pos = self.find_max_cell_location(field, finest_levels) mg, mc, mv, pos = self.find_max_cell_location(field, finest_levels) File "/homeappl/home/regan/appl_taito/YT/yt-x86_64/src/yt-hg/yt/data_objects/object_finding_mixin.py", line 66, in find_max_cell_location mg, mc, mv, pos = self.find_max_cell_location(field, finest_levels) File "/homeappl/home/regan/appl_taito/YT/yt-x86_64/src/yt-hg/yt/data_objects/object_finding_mixin.py", line 66, in find_max_cell_location File "/homeappl/home/regan/appl_taito/YT/yt-x86_64/src/yt-hg/yt/data_objects/object_finding_mixin.py", line 66, in find_max_cell_location mg, mc, mv, pos = self.find_max_cell_location(field, finest_levels)
yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
Hi,
It seems that the mpi4py installation is not the issue. I reinstalled as Nathan suggested and got the same errors with the test script.
If anyone has other suggestions as to why this is failing let me know!
Thanks!
John
On Sun, Dec 29, 2013 at 10:29 PM, Nathan Goldbaum nathan12343@gmail.comwrote:
yt itself does not link against an mpi library. Instead, we use mpi4py to handle bindings to the local mpi installation. To be sure that mpi4py is linked against the correct mpi lbirary, you should first uninstall mpi4py:
$ pip uninstall mpi4py
then download an mpi4py tarball, unzip it, and build it using the correct sysem MPI
$ wget https://mpi4py.googlecode.com/files/mpi4py-1.3.1.tar.gz $ tar xzvf mpi4py-1.3.1.tar.gz $ python setup.py build --mpicc=/where/you/have/mpicc $ python setup.py install
When you buid mpi4py, be sure to specify the path to the MPI compiler for the system MPI package you want to link against. For more detail on building mpi4py, check here: http://mpi4py.scipy.org/docs/usrman/install.html#building
On Sun, Dec 29, 2013 at 11:54 AM, John Regan johnanthonyregan@gmail.comwrote:
Hi Sam,
I'm running the script using srun pyyt ptest.py --parallel
That could well be picking up the system mpirun rather than the mpi that YT was built with - I hadn't considered that. I'll check that. The traceback is at the link below.
http://paste.yt-project.org/show/4174/
The version of YT is the dev version - that's what's sourced when the batch script is run. I've tried with the stable version as well and got the same result. I think you might be on to something with the mpirun library issue though.
Thanks for the tip!
John
(yt-x86_64)[regan@taito-login3 Projections]$ yt instinfo
yt module located at: /homeappl/home/regan/appl_taito/YT/yt-x86_64/src/yt-hg The supplemental repositories are located at: /homeappl/home/regan/appl_taito/YT/yt-x86_64/src/yt-supplemental
The current version of the code is:
ec9a5117fc08
This installation CAN be automatically updated. yt dependencies were last updated on Tue Dec 3 17:12:02 EET 2013
To update all dependencies, run "yt update --all".
On Sun, Dec 29, 2013 at 8:22 PM, Sam Skillman samskillman@gmail.comwrote:
Hi John,
Can you paste the full traceback using: http://paste.yt-project.org? Also, are you running with mpirun python your_script.py --parallel? The only thought I have at the moment is that there is a mismatch between the MPI you are running mpirun with and the MPI that mpi4py was built with.
Also, which version of yt are you using (the results of "yt instinfo")?
Thanks, Sam
On Sat, Dec 28, 2013 at 11:36 PM, John Regan <johnanthonyregan@gmail.com
wrote:
Hi All,
I've started to see a strange error in a simple enough parallel computation.
I can reproduce it with the script below but with some other scripts where the code is embedded it runs fine so it may be a race condition.
from yt.mods import *
pf = load("/wrk/regan/H2/Ctrl/DD0035/Data0035") v, c = pf.h.find_max("Density") print v, c
Thanks in advance for any help!
Cheers, John
Error:
Traceback (most recent call last): P001 yt : [INFO ] 2013-12-29 09:31:27,756 Max Value is 3.67832e-16 at 0.5025423020124483 0.4896353334188484 0.4947452098131160 in grid EnzoGrid_1214 at level 18 (3, 3, 2) Traceback (most recent call last): File "ptest.py", line 5, in <module> P003 yt : [INFO ] 2013-12-29 09:31:27,756 Max Value is 3.67832e-16 at 0.5025423020124483 0.4896353334188484 0.4947452098131160 in grid EnzoGrid_1214 at level 18 (3, 3, 2) File "ptest.py", line 5, in <module> Traceback (most recent call last): File "ptest.py", line 5, in <module> P002 yt : [INFO ] 2013-12-29 09:31:27,756 Max Value is 3.67832e-16 at 0.5025423020124483 0.4896353334188484 0.4947452098131160 in grid EnzoGrid_1214 at level 18 (3, 3, 2) Traceback (most recent call last): P004 yt : [INFO ] 2013-12-29 09:31:27,756 Max Value is 3.67832e-16 at 0.5025423020124483 0.4896353334188484 0.4947452098131160 in grid EnzoGrid_1214 at level 18 (3, 3, 2) Traceback (most recent call last): File "ptest.py", line 5, in <module> P005 yt : [INFO ] 2013-12-29 09:31:27,756 Max Value is 3.67832e-16 at 0.5025423020124483 0.4896353334188484 0.4947452098131160 in grid EnzoGrid_1214 at level 18 (3, 3, 2) Traceback (most recent call last): File "ptest.py", line 5, in <module> P006 yt : [INFO ] 2013-12-29 09:31:27,756 Max Value is 3.67832e-16 at 0.5025423020124483 0.4896353334188484 0.4947452098131160 in grid EnzoGrid_1214 at level 18 (3, 3, 2) Traceback (most recent call last): File "ptest.py", line 5, in <module> Traceback (most recent call last): File "ptest.py", line 5, in <module> Traceback (most recent call last): File "ptest.py", line 5, in <module> File "ptest.py", line 5, in <module> v, c = pf.h.find_max("Density") v, c = pf.h.find_max("Density") v, c = pf.h.find_max("Density") v, c = pf.h.find_max("Density") File "/homeappl/home/regan/appl_taito/YT/yt-x86_64/src/yt-hg/yt/data_objects/object_finding_mixin.py", line 51, in find_max v, c = pf.h.find_max("Density") File "/homeappl/home/regan/appl_taito/YT/yt-x86_64/src/yt-hg/yt/data_objects/object_finding_mixin.py", line 51, in find_max File "/homeappl/home/regan/appl_taito/YT/yt-x86_64/src/yt-hg/yt/data_objects/object_finding_mixin.py", line 51, in find_max File "/homeappl/home/regan/appl_taito/YT/yt-x86_64/src/yt-hg/yt/data_objects/object_finding_mixin.py", line 51, in find_max v, c = pf.h.find_max("Density") File "/homeappl/home/regan/appl_taito/YT/yt-x86_64/src/yt-hg/yt/data_objects/object_finding_mixin.py", line 51, in find_max v, c = pf.h.find_max("Density") v, c = pf.h.find_max("Density") v, c = pf.h.find_max("Density") File "/homeappl/home/regan/appl_taito/YT/yt-x86_64/src/yt-hg/yt/data_objects/object_finding_mixin.py", line 51, in find_max File "/homeappl/home/regan/appl_taito/YT/yt-x86_64/src/yt-hg/yt/data_objects/object_finding_mixin.py", line 51, in find_max File "/homeappl/home/regan/appl_taito/YT/yt-x86_64/src/yt-hg/yt/data_objects/object_finding_mixin.py", line 51, in find_max File "/homeappl/home/regan/appl_taito/YT/yt-x86_64/src/yt-hg/yt/data_objects/object_finding_mixin.py", line 51, in find_max mg, mc, mv, pos = self.find_max_cell_location(field, finest_levels) mg, mc, mv, pos = self.find_max_cell_location(field, finest_levels) File "/homeappl/home/regan/appl_taito/YT/yt-x86_64/src/yt-hg/yt/data_objects/object_finding_mixin.py", line 66, in find_max_cell_location mg, mc, mv, pos = self.find_max_cell_location(field, finest_levels) File "/homeappl/home/regan/appl_taito/YT/yt-x86_64/src/yt-hg/yt/data_objects/object_finding_mixin.py", line 66, in find_max_cell_location File "/homeappl/home/regan/appl_taito/YT/yt-x86_64/src/yt-hg/yt/data_objects/object_finding_mixin.py", line 66, in find_max_cell_location mg, mc, mv, pos = self.find_max_cell_location(field, finest_levels)
yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
-
John Regan -
Nathan Goldbaum -
Sam Skillman