Hi, Everybody!
I'm trying to run a parallel projection on a 1024^3 simulation, but I keep getting out-of-memory. Serial works fine. This is all on Stampede, on an interactive node. My script is direct from the website, and is below. Is there something dumb I'm doing?
Thanks!
I do
$ibrun -np 2 python parallel_test.py
and parallel_test.py contains """ from mpi4py import MPI import yt yt.enable_parallelism()
ds = yt.load("/scratch/00369/tg456484/Paper49d_moresims/ze01_M10_MA1_1024_quan/DD0095/data0095") /DD%04d/data%04d"%(22,22)) p = yt.ProjectionPlot(ds, "x", "density") p.save() """
my output then looks like:
000 yt : [INFO ] 2019-02-22 13:00:41,879 Gathering a field list (this may take a moment.) File "parallel_test.py", line 13, in <module> p = yt.ProjectionPlot(ds, "x", "density") File "/home1/00369/tg456484/local-yt-2019-02-22-py3/yt-conda/lib/python3.7/site-packages/yt/visualizati on/plot_window.py", line 1480, in __init__ max_level=max_level) File "/home1/00369/tg456484/local-yt-2019-02-22-py3/yt-conda/lib/python3.7/site-packages/yt/data_object s/construction_data_containers.py", line 270, in __init__ self.get_data(field) File "/home1/00369/tg456484/local-yt-2019-02-22-py3/yt-conda/lib/python3.7/site-packages/yt/data_object s/construction_data_containers.py", line 334, in get_data self._initialize_chunk(chunk, tree)
File "/home1/00369/tg456484/local-yt-2019-02-22-py3/yt-conda/lib/python3.7/site-packages/yt/geometry/geometry_handler.py", line 271, in cached_func tr = func(self) File "/home1/00369/tg456484/local-yt-2019-02-22-py3/yt-conda/lib/python3.7/site-packages/yt/geometry/geometry_handler.py", line 332, in icoords ci = np.empty((self.data_size, 3), dtype='int64') P006 yt : [ERROR ] 2019-02-22 13:03:36,145 MemoryError: File "parallel_test.py", line 13, in <module> p = yt.ProjectionPlot(ds, "x", "density") File "/home1/00369/tg456484/local-yt-2019-02-22-py3/yt-conda/lib/python3.7/site-packages/yt/visualization/plot_window.py", line 1480, in __init__ max_level=max_level) File "/home1/00369/tg456484/local-yt-2019-02-22-py3/yt-conda/lib/python3.7/site-packages/yt/data_objects/construction_data_containers.py", line 270, in __init__ self.get_data(field) File "/home1/00369/tg456484/local-yt-2019-02-22-py3/yt-conda/lib/python3.7/site-packages/yt/data_objects/construction_data_containers.py", line 334, in get_data self._initialize_chunk(chunk, tree) File "/home1/00369/tg456484/local-yt-2019-02-22-py3/yt-conda/lib/python3.7/site-packages/yt/data_objects/construction_data_containers.py", line 401, in _initialize_chunk icoords = chunk.icoords File "/home1/00369/tg456484/local-yt-2019-02-22-py3/yt-conda/lib/python3.7/site-packages/yt/data_objects/data_containers.py", line 1555, in icoords return self._current_chunk.icoords File "/home1/00369/tg456484/local-yt-2019-02-22-py3/yt-conda/lib/python3.7/site-packages/yt/geometry/geometry_handler.py", line 271, in cached_func tr = func(self) File "/home1/00369/tg456484/local-yt-2019-02-22-py3/yt-conda/lib/python3.7/site-packages/yt/geometry/geometry_handler.py", line 332, in icoords ci = np.empty((self.data_size, 3), dtype='int64') P007 yt : [ERROR ] 2019-02-22 13:03:36,145 MemoryError: P002 yt : [ERROR ] 2019-02-22 13:03:36,200 Error occured on rank 2. P004 yt : [ERROR ] 2019-02-22 13:03:36,200 Error occured on rank 4. P000 yt : [ERROR ] 2019-02-22 13:03:36,200 Error occured on rank 0. P001 yt : [ERROR ] 2019-02-22 13:03:36,200 Error occured on rank 1. P003 yt : [ERROR ] 2019-02-22 13:03:36,200 Error occured on rank 3. P005 yt : [ERROR ] 2019-02-22 13:03:36,200 Error occured on rank 5. P006 yt : [ERROR ] 2019-02-22 13:03:36,200 Error occured on rank 6. P007 yt : [ERROR ] 2019-02-22 13:03:36,200 Error occured on rank 7. application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0 application called MPI_Abort(MPI_COMM_WORLD, 1) - process 1 application called MPI_Abort(MPI_COMM_WORLD, 1) - process 2 application called MPI_Abort(MPI_COMM_WORLD, 1) - process 3 application called MPI_Abort(MPI_COMM_WORLD, 1) - process 5 application called MPI_Abort(MPI_COMM_WORLD, 1) - process 6 application called MPI_Abort(MPI_COMM_WORLD, 1) - process 7 application called MPI_Abort(MPI_COMM_WORLD, 1) - process 4 TACC: MPI job exited with code: 1 TACC: Shutdown complete. Exiting.
Can you give more detail about the data? Enzo I'm guessing? Unigrid or AMR? How many grid patches in the output?
My first thought is that this is probably a bug, especially if it runs within memory bounds on one core, there's probably some unnecessary double-allocation going on.
On Fri, Feb 22, 2019 at 1:31 PM David Collins dcollins4096@gmail.com wrote:
Hi, Everybody!
I'm trying to run a parallel projection on a 1024^3 simulation, but I keep getting out-of-memory. Serial works fine. This is all on Stampede, on an interactive node. My script is direct from the website, and is below. Is there something dumb I'm doing?
Thanks!
I do
$ibrun -np 2 python parallel_test.py
and parallel_test.py contains """ from mpi4py import MPI import yt yt.enable_parallelism()
ds = yt.load("/scratch/00369/tg456484/Paper49d_moresims/ze01_M10_MA1_1024_quan/DD0095/data0095") /DD%04d/data%04d"%(22,22)) p = yt.ProjectionPlot(ds, "x", "density") p.save() """
my output then looks like:
000 yt : [INFO ] 2019-02-22 13:00:41,879 Gathering a field list (this may take a moment.) File "parallel_test.py", line 13, in <module> p = yt.ProjectionPlot(ds, "x", "density") File "/home1/00369/tg456484/local-yt-2019-02-22-py3/yt-conda/lib/python3.7/site-packages/yt/visualizati on/plot_window.py", line 1480, in __init__ max_level=max_level) File "/home1/00369/tg456484/local-yt-2019-02-22-py3/yt-conda/lib/python3.7/site-packages/yt/data_object s/construction_data_containers.py", line 270, in __init__ self.get_data(field) File "/home1/00369/tg456484/local-yt-2019-02-22-py3/yt-conda/lib/python3.7/site-packages/yt/data_object s/construction_data_containers.py", line 334, in get_data self._initialize_chunk(chunk, tree)
File "/home1/00369/tg456484/local-yt-2019-02-22-py3/yt-conda/lib/python3.7/site-packages/yt/geometry/geometry_handler.py", line 271, in cached_func tr = func(self) File "/home1/00369/tg456484/local-yt-2019-02-22-py3/yt-conda/lib/python3.7/site-packages/yt/geometry/geometry_handler.py", line 332, in icoords ci = np.empty((self.data_size, 3), dtype='int64') P006 yt : [ERROR ] 2019-02-22 13:03:36,145 MemoryError: File "parallel_test.py", line 13, in <module> p = yt.ProjectionPlot(ds, "x", "density") File "/home1/00369/tg456484/local-yt-2019-02-22-py3/yt-conda/lib/python3.7/site-packages/yt/visualization/plot_window.py", line 1480, in __init__ max_level=max_level) File "/home1/00369/tg456484/local-yt-2019-02-22-py3/yt-conda/lib/python3.7/site-packages/yt/data_objects/construction_data_containers.py", line 270, in __init__ self.get_data(field) File "/home1/00369/tg456484/local-yt-2019-02-22-py3/yt-conda/lib/python3.7/site-packages/yt/data_objects/construction_data_containers.py", line 334, in get_data self._initialize_chunk(chunk, tree) File "/home1/00369/tg456484/local-yt-2019-02-22-py3/yt-conda/lib/python3.7/site-packages/yt/data_objects/construction_data_containers.py", line 401, in _initialize_chunk icoords = chunk.icoords File "/home1/00369/tg456484/local-yt-2019-02-22-py3/yt-conda/lib/python3.7/site-packages/yt/data_objects/data_containers.py", line 1555, in icoords return self._current_chunk.icoords File "/home1/00369/tg456484/local-yt-2019-02-22-py3/yt-conda/lib/python3.7/site-packages/yt/geometry/geometry_handler.py", line 271, in cached_func tr = func(self) File "/home1/00369/tg456484/local-yt-2019-02-22-py3/yt-conda/lib/python3.7/site-packages/yt/geometry/geometry_handler.py", line 332, in icoords ci = np.empty((self.data_size, 3), dtype='int64') P007 yt : [ERROR ] 2019-02-22 13:03:36,145 MemoryError: P002 yt : [ERROR ] 2019-02-22 13:03:36,200 Error occured on rank 2. P004 yt : [ERROR ] 2019-02-22 13:03:36,200 Error occured on rank 4. P000 yt : [ERROR ] 2019-02-22 13:03:36,200 Error occured on rank 0. P001 yt : [ERROR ] 2019-02-22 13:03:36,200 Error occured on rank 1. P003 yt : [ERROR ] 2019-02-22 13:03:36,200 Error occured on rank 3. P005 yt : [ERROR ] 2019-02-22 13:03:36,200 Error occured on rank 5. P006 yt : [ERROR ] 2019-02-22 13:03:36,200 Error occured on rank 6. P007 yt : [ERROR ] 2019-02-22 13:03:36,200 Error occured on rank 7. application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0 application called MPI_Abort(MPI_COMM_WORLD, 1) - process 1 application called MPI_Abort(MPI_COMM_WORLD, 1) - process 2 application called MPI_Abort(MPI_COMM_WORLD, 1) - process 3 application called MPI_Abort(MPI_COMM_WORLD, 1) - process 5 application called MPI_Abort(MPI_COMM_WORLD, 1) - process 6 application called MPI_Abort(MPI_COMM_WORLD, 1) - process 7 application called MPI_Abort(MPI_COMM_WORLD, 1) - process 4 TACC: MPI job exited with code: 1 TACC: Shutdown complete. Exiting.
-- -- Sent from a computer. _______________________________________________ yt-users mailing list -- yt-users@python.org To unsubscribe send an email to yt-users-leave@python.org
Hi--
It's a Enzo run, unigrid at 1024^3, and I ran on 4096 cores, so 4096 patches. No particles, MHD with Dedner and Stochastic forcing, no other physics. Thanks, d.
On Fri, Feb 22, 2019 at 2:37 PM Nathan Goldbaum nathan12343@gmail.com wrote:
Can you give more detail about the data? Enzo I'm guessing? Unigrid or AMR? How many grid patches in the output?
My first thought is that this is probably a bug, especially if it runs within memory bounds on one core, there's probably some unnecessary double-allocation going on.
On Fri, Feb 22, 2019 at 1:31 PM David Collins dcollins4096@gmail.com wrote:
Hi, Everybody!
I'm trying to run a parallel projection on a 1024^3 simulation, but I keep getting out-of-memory. Serial works fine. This is all on Stampede, on an interactive node. My script is direct from the website, and is below. Is there something dumb I'm doing?
Thanks!
I do
$ibrun -np 2 python parallel_test.py
and parallel_test.py contains """ from mpi4py import MPI import yt yt.enable_parallelism()
ds = yt.load("/scratch/00369/tg456484/Paper49d_moresims/ze01_M10_MA1_1024_quan/DD0095/data0095") /DD%04d/data%04d"%(22,22)) p = yt.ProjectionPlot(ds, "x", "density") p.save() """
my output then looks like:
000 yt : [INFO ] 2019-02-22 13:00:41,879 Gathering a field list (this may take a moment.) File "parallel_test.py", line 13, in <module> p = yt.ProjectionPlot(ds, "x", "density") File "/home1/00369/tg456484/local-yt-2019-02-22-py3/yt-conda/lib/python3.7/site-packages/yt/visualizati on/plot_window.py", line 1480, in __init__ max_level=max_level) File "/home1/00369/tg456484/local-yt-2019-02-22-py3/yt-conda/lib/python3.7/site-packages/yt/data_object s/construction_data_containers.py", line 270, in __init__ self.get_data(field) File "/home1/00369/tg456484/local-yt-2019-02-22-py3/yt-conda/lib/python3.7/site-packages/yt/data_object s/construction_data_containers.py", line 334, in get_data self._initialize_chunk(chunk, tree)
File "/home1/00369/tg456484/local-yt-2019-02-22-py3/yt-conda/lib/python3.7/site-packages/yt/geometry/geometry_handler.py", line 271, in cached_func tr = func(self) File "/home1/00369/tg456484/local-yt-2019-02-22-py3/yt-conda/lib/python3.7/site-packages/yt/geometry/geometry_handler.py", line 332, in icoords ci = np.empty((self.data_size, 3), dtype='int64') P006 yt : [ERROR ] 2019-02-22 13:03:36,145 MemoryError: File "parallel_test.py", line 13, in <module> p = yt.ProjectionPlot(ds, "x", "density") File "/home1/00369/tg456484/local-yt-2019-02-22-py3/yt-conda/lib/python3.7/site-packages/yt/visualization/plot_window.py", line 1480, in __init__ max_level=max_level) File "/home1/00369/tg456484/local-yt-2019-02-22-py3/yt-conda/lib/python3.7/site-packages/yt/data_objects/construction_data_containers.py", line 270, in __init__ self.get_data(field) File "/home1/00369/tg456484/local-yt-2019-02-22-py3/yt-conda/lib/python3.7/site-packages/yt/data_objects/construction_data_containers.py", line 334, in get_data self._initialize_chunk(chunk, tree) File "/home1/00369/tg456484/local-yt-2019-02-22-py3/yt-conda/lib/python3.7/site-packages/yt/data_objects/construction_data_containers.py", line 401, in _initialize_chunk icoords = chunk.icoords File "/home1/00369/tg456484/local-yt-2019-02-22-py3/yt-conda/lib/python3.7/site-packages/yt/data_objects/data_containers.py", line 1555, in icoords return self._current_chunk.icoords File "/home1/00369/tg456484/local-yt-2019-02-22-py3/yt-conda/lib/python3.7/site-packages/yt/geometry/geometry_handler.py", line 271, in cached_func tr = func(self) File "/home1/00369/tg456484/local-yt-2019-02-22-py3/yt-conda/lib/python3.7/site-packages/yt/geometry/geometry_handler.py", line 332, in icoords ci = np.empty((self.data_size, 3), dtype='int64') P007 yt : [ERROR ] 2019-02-22 13:03:36,145 MemoryError: P002 yt : [ERROR ] 2019-02-22 13:03:36,200 Error occured on rank 2. P004 yt : [ERROR ] 2019-02-22 13:03:36,200 Error occured on rank 4. P000 yt : [ERROR ] 2019-02-22 13:03:36,200 Error occured on rank 0. P001 yt : [ERROR ] 2019-02-22 13:03:36,200 Error occured on rank 1. P003 yt : [ERROR ] 2019-02-22 13:03:36,200 Error occured on rank 3. P005 yt : [ERROR ] 2019-02-22 13:03:36,200 Error occured on rank 5. P006 yt : [ERROR ] 2019-02-22 13:03:36,200 Error occured on rank 6. P007 yt : [ERROR ] 2019-02-22 13:03:36,200 Error occured on rank 7. application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0 application called MPI_Abort(MPI_COMM_WORLD, 1) - process 1 application called MPI_Abort(MPI_COMM_WORLD, 1) - process 2 application called MPI_Abort(MPI_COMM_WORLD, 1) - process 3 application called MPI_Abort(MPI_COMM_WORLD, 1) - process 5 application called MPI_Abort(MPI_COMM_WORLD, 1) - process 6 application called MPI_Abort(MPI_COMM_WORLD, 1) - process 7 application called MPI_Abort(MPI_COMM_WORLD, 1) - process 4 TACC: MPI job exited with code: 1 TACC: Shutdown complete. Exiting.
-- -- Sent from a computer. _______________________________________________ yt-users mailing list -- yt-users@python.org To unsubscribe send an email to yt-users-leave@python.org
yt-users mailing list -- yt-users@python.org To unsubscribe send an email to yt-users-leave@python.org
OK, it would be great if you could file an issue about this. If possible please include a link to a copy of the dataset so that one of us can try to reproduce the issue.
On Fri, Feb 22, 2019 at 3:26 PM David Collins dcollins4096@gmail.com wrote:
Hi--
It's a Enzo run, unigrid at 1024^3, and I ran on 4096 cores, so 4096 patches. No particles, MHD with Dedner and Stochastic forcing, no other physics. Thanks, d.
On Fri, Feb 22, 2019 at 2:37 PM Nathan Goldbaum nathan12343@gmail.com wrote:
Can you give more detail about the data? Enzo I'm guessing? Unigrid or AMR? How many grid patches in the output?
My first thought is that this is probably a bug, especially if it runs within memory bounds on one core, there's probably some unnecessary double-allocation going on.
On Fri, Feb 22, 2019 at 1:31 PM David Collins dcollins4096@gmail.com wrote:
Hi, Everybody!
I'm trying to run a parallel projection on a 1024^3 simulation, but I keep getting out-of-memory. Serial works fine. This is all on Stampede, on an interactive node. My script is direct from the website, and is below. Is there something dumb I'm doing?
Thanks!
I do
$ibrun -np 2 python parallel_test.py
and parallel_test.py contains """ from mpi4py import MPI import yt yt.enable_parallelism()
ds = yt.load("/scratch/00369/tg456484/Paper49d_moresims/ze01_M10_MA1_1024_quan/DD0095/data0095") /DD%04d/data%04d"%(22,22)) p = yt.ProjectionPlot(ds, "x", "density") p.save() """
my output then looks like:
000 yt : [INFO ] 2019-02-22 13:00:41,879 Gathering a field list (this may take a moment.) File "parallel_test.py", line 13, in <module> p = yt.ProjectionPlot(ds, "x", "density") File "/home1/00369/tg456484/local-yt-2019-02-22-py3/yt-conda/lib/python3.7/site-packages/yt/visualizati on/plot_window.py", line 1480, in __init__ max_level=max_level) File "/home1/00369/tg456484/local-yt-2019-02-22-py3/yt-conda/lib/python3.7/site-packages/yt/data_object s/construction_data_containers.py", line 270, in __init__ self.get_data(field) File "/home1/00369/tg456484/local-yt-2019-02-22-py3/yt-conda/lib/python3.7/site-packages/yt/data_object s/construction_data_containers.py", line 334, in get_data self._initialize_chunk(chunk, tree)
File "/home1/00369/tg456484/local-yt-2019-02-22-py3/yt-conda/lib/python3.7/site-packages/yt/geometry/geometry_handler.py", line 271, in cached_func tr = func(self) File "/home1/00369/tg456484/local-yt-2019-02-22-py3/yt-conda/lib/python3.7/site-packages/yt/geometry/geometry_handler.py", line 332, in icoords ci = np.empty((self.data_size, 3), dtype='int64') P006 yt : [ERROR ] 2019-02-22 13:03:36,145 MemoryError: File "parallel_test.py", line 13, in <module> p = yt.ProjectionPlot(ds, "x", "density") File "/home1/00369/tg456484/local-yt-2019-02-22-py3/yt-conda/lib/python3.7/site-packages/yt/visualization/plot_window.py", line 1480, in __init__ max_level=max_level) File "/home1/00369/tg456484/local-yt-2019-02-22-py3/yt-conda/lib/python3.7/site-packages/yt/data_objects/construction_data_containers.py", line 270, in __init__ self.get_data(field) File "/home1/00369/tg456484/local-yt-2019-02-22-py3/yt-conda/lib/python3.7/site-packages/yt/data_objects/construction_data_containers.py", line 334, in get_data self._initialize_chunk(chunk, tree) File "/home1/00369/tg456484/local-yt-2019-02-22-py3/yt-conda/lib/python3.7/site-packages/yt/data_objects/construction_data_containers.py", line 401, in _initialize_chunk icoords = chunk.icoords File "/home1/00369/tg456484/local-yt-2019-02-22-py3/yt-conda/lib/python3.7/site-packages/yt/data_objects/data_containers.py", line 1555, in icoords return self._current_chunk.icoords File "/home1/00369/tg456484/local-yt-2019-02-22-py3/yt-conda/lib/python3.7/site-packages/yt/geometry/geometry_handler.py", line 271, in cached_func tr = func(self) File "/home1/00369/tg456484/local-yt-2019-02-22-py3/yt-conda/lib/python3.7/site-packages/yt/geometry/geometry_handler.py", line 332, in icoords ci = np.empty((self.data_size, 3), dtype='int64') P007 yt : [ERROR ] 2019-02-22 13:03:36,145 MemoryError: P002 yt : [ERROR ] 2019-02-22 13:03:36,200 Error occured on rank 2. P004 yt : [ERROR ] 2019-02-22 13:03:36,200 Error occured on rank 4. P000 yt : [ERROR ] 2019-02-22 13:03:36,200 Error occured on rank 0. P001 yt : [ERROR ] 2019-02-22 13:03:36,200 Error occured on rank 1. P003 yt : [ERROR ] 2019-02-22 13:03:36,200 Error occured on rank 3. P005 yt : [ERROR ] 2019-02-22 13:03:36,200 Error occured on rank 5. P006 yt : [ERROR ] 2019-02-22 13:03:36,200 Error occured on rank 6. P007 yt : [ERROR ] 2019-02-22 13:03:36,200 Error occured on rank 7. application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0 application called MPI_Abort(MPI_COMM_WORLD, 1) - process 1 application called MPI_Abort(MPI_COMM_WORLD, 1) - process 2 application called MPI_Abort(MPI_COMM_WORLD, 1) - process 3 application called MPI_Abort(MPI_COMM_WORLD, 1) - process 5 application called MPI_Abort(MPI_COMM_WORLD, 1) - process 6 application called MPI_Abort(MPI_COMM_WORLD, 1) - process 7 application called MPI_Abort(MPI_COMM_WORLD, 1) - process 4 TACC: MPI job exited with code: 1 TACC: Shutdown complete. Exiting.
-- -- Sent from a computer. _______________________________________________ yt-users mailing list -- yt-users@python.org To unsubscribe send an email to yt-users-leave@python.org
yt-users mailing list -- yt-users@python.org To unsubscribe send an email to yt-users-leave@python.org
-- -- Sent from a computer. _______________________________________________ yt-users mailing list -- yt-users@python.org To unsubscribe send an email to yt-users-leave@python.org
I will try to make a mock dataset. My *guess* is that this is related to the issue Andrew Myers was having, and his hotfix might work for this.
On Fri, Feb 22, 2019 at 3:30 PM Nathan Goldbaum nathan12343@gmail.com wrote:
OK, it would be great if you could file an issue about this. If possible please include a link to a copy of the dataset so that one of us can try to reproduce the issue.
On Fri, Feb 22, 2019 at 3:26 PM David Collins dcollins4096@gmail.com wrote:
Hi--
It's a Enzo run, unigrid at 1024^3, and I ran on 4096 cores, so 4096 patches. No particles, MHD with Dedner and Stochastic forcing, no other physics. Thanks, d.
On Fri, Feb 22, 2019 at 2:37 PM Nathan Goldbaum nathan12343@gmail.com wrote:
Can you give more detail about the data? Enzo I'm guessing? Unigrid or AMR? How many grid patches in the output?
My first thought is that this is probably a bug, especially if it runs within memory bounds on one core, there's probably some unnecessary double-allocation going on.
On Fri, Feb 22, 2019 at 1:31 PM David Collins dcollins4096@gmail.com wrote:
Hi, Everybody!
I'm trying to run a parallel projection on a 1024^3 simulation, but I keep getting out-of-memory. Serial works fine. This is all on Stampede, on an interactive node. My script is direct from the website, and is below. Is there something dumb I'm doing?
Thanks!
I do
$ibrun -np 2 python parallel_test.py
and parallel_test.py contains """ from mpi4py import MPI import yt yt.enable_parallelism()
ds = yt.load("/scratch/00369/tg456484/Paper49d_moresims/ze01_M10_MA1_1024_quan/DD0095/data0095") /DD%04d/data%04d"%(22,22)) p = yt.ProjectionPlot(ds, "x", "density") p.save() """
my output then looks like:
000 yt : [INFO ] 2019-02-22 13:00:41,879 Gathering a field list (this may take a moment.) File "parallel_test.py", line 13, in <module> p = yt.ProjectionPlot(ds, "x", "density") File "/home1/00369/tg456484/local-yt-2019-02-22-py3/yt-conda/lib/python3.7/site-packages/yt/visualizati on/plot_window.py", line 1480, in __init__ max_level=max_level) File "/home1/00369/tg456484/local-yt-2019-02-22-py3/yt-conda/lib/python3.7/site-packages/yt/data_object s/construction_data_containers.py", line 270, in __init__ self.get_data(field) File "/home1/00369/tg456484/local-yt-2019-02-22-py3/yt-conda/lib/python3.7/site-packages/yt/data_object s/construction_data_containers.py", line 334, in get_data self._initialize_chunk(chunk, tree)
File "/home1/00369/tg456484/local-yt-2019-02-22-py3/yt-conda/lib/python3.7/site-packages/yt/geometry/geometry_handler.py", line 271, in cached_func tr = func(self) File "/home1/00369/tg456484/local-yt-2019-02-22-py3/yt-conda/lib/python3.7/site-packages/yt/geometry/geometry_handler.py", line 332, in icoords ci = np.empty((self.data_size, 3), dtype='int64') P006 yt : [ERROR ] 2019-02-22 13:03:36,145 MemoryError: File "parallel_test.py", line 13, in <module> p = yt.ProjectionPlot(ds, "x", "density") File "/home1/00369/tg456484/local-yt-2019-02-22-py3/yt-conda/lib/python3.7/site-packages/yt/visualization/plot_window.py", line 1480, in __init__ max_level=max_level) File "/home1/00369/tg456484/local-yt-2019-02-22-py3/yt-conda/lib/python3.7/site-packages/yt/data_objects/construction_data_containers.py", line 270, in __init__ self.get_data(field) File "/home1/00369/tg456484/local-yt-2019-02-22-py3/yt-conda/lib/python3.7/site-packages/yt/data_objects/construction_data_containers.py", line 334, in get_data self._initialize_chunk(chunk, tree) File "/home1/00369/tg456484/local-yt-2019-02-22-py3/yt-conda/lib/python3.7/site-packages/yt/data_objects/construction_data_containers.py", line 401, in _initialize_chunk icoords = chunk.icoords File "/home1/00369/tg456484/local-yt-2019-02-22-py3/yt-conda/lib/python3.7/site-packages/yt/data_objects/data_containers.py", line 1555, in icoords return self._current_chunk.icoords File "/home1/00369/tg456484/local-yt-2019-02-22-py3/yt-conda/lib/python3.7/site-packages/yt/geometry/geometry_handler.py", line 271, in cached_func tr = func(self) File "/home1/00369/tg456484/local-yt-2019-02-22-py3/yt-conda/lib/python3.7/site-packages/yt/geometry/geometry_handler.py", line 332, in icoords ci = np.empty((self.data_size, 3), dtype='int64') P007 yt : [ERROR ] 2019-02-22 13:03:36,145 MemoryError: P002 yt : [ERROR ] 2019-02-22 13:03:36,200 Error occured on rank 2. P004 yt : [ERROR ] 2019-02-22 13:03:36,200 Error occured on rank 4. P000 yt : [ERROR ] 2019-02-22 13:03:36,200 Error occured on rank 0. P001 yt : [ERROR ] 2019-02-22 13:03:36,200 Error occured on rank 1. P003 yt : [ERROR ] 2019-02-22 13:03:36,200 Error occured on rank 3. P005 yt : [ERROR ] 2019-02-22 13:03:36,200 Error occured on rank 5. P006 yt : [ERROR ] 2019-02-22 13:03:36,200 Error occured on rank 6. P007 yt : [ERROR ] 2019-02-22 13:03:36,200 Error occured on rank 7. application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0 application called MPI_Abort(MPI_COMM_WORLD, 1) - process 1 application called MPI_Abort(MPI_COMM_WORLD, 1) - process 2 application called MPI_Abort(MPI_COMM_WORLD, 1) - process 3 application called MPI_Abort(MPI_COMM_WORLD, 1) - process 5 application called MPI_Abort(MPI_COMM_WORLD, 1) - process 6 application called MPI_Abort(MPI_COMM_WORLD, 1) - process 7 application called MPI_Abort(MPI_COMM_WORLD, 1) - process 4 TACC: MPI job exited with code: 1 TACC: Shutdown complete. Exiting.
-- -- Sent from a computer. _______________________________________________ yt-users mailing list -- yt-users@python.org To unsubscribe send an email to yt-users-leave@python.org
yt-users mailing list -- yt-users@python.org To unsubscribe send an email to yt-users-leave@python.org
-- -- Sent from a computer. _______________________________________________ yt-users mailing list -- yt-users@python.org To unsubscribe send an email to yt-users-leave@python.org
yt-users mailing list -- yt-users@python.org To unsubscribe send an email to yt-users-leave@python.org
Grand, thanks, Matt. If you're on stampede, any of the data in /scratch/00369/tg456484/Paper49d_moresims/ze01_M10_MA1_1024_quan should be readable. I've made a ticket, as well. The set was made with the StochasticForcing parameter file in the run directory with DualEnergy off.
Thanks, and let me know if there's anything else you need from me! d.
On Fri, Feb 22, 2019 at 4:38 PM Matthew Turk matthewturk@gmail.com wrote:
I will try to make a mock dataset. My *guess* is that this is related to the issue Andrew Myers was having, and his hotfix might work for this.
On Fri, Feb 22, 2019 at 3:30 PM Nathan Goldbaum nathan12343@gmail.com wrote:
OK, it would be great if you could file an issue about this. If possible please include a link to a copy of the dataset so that one of us can try to reproduce the issue.
On Fri, Feb 22, 2019 at 3:26 PM David Collins dcollins4096@gmail.com wrote:
Hi--
It's a Enzo run, unigrid at 1024^3, and I ran on 4096 cores, so 4096 patches. No particles, MHD with Dedner and Stochastic forcing, no other physics. Thanks, d.
On Fri, Feb 22, 2019 at 2:37 PM Nathan Goldbaum nathan12343@gmail.com wrote:
Can you give more detail about the data? Enzo I'm guessing? Unigrid or AMR? How many grid patches in the output?
My first thought is that this is probably a bug, especially if it runs within memory bounds on one core, there's probably some unnecessary double-allocation going on.
On Fri, Feb 22, 2019 at 1:31 PM David Collins dcollins4096@gmail.com wrote:
Hi, Everybody!
I'm trying to run a parallel projection on a 1024^3 simulation, but I keep getting out-of-memory. Serial works fine. This is all on Stampede, on an interactive node. My script is direct from the website, and is below. Is there something dumb I'm doing?
Thanks!
I do
$ibrun -np 2 python parallel_test.py
and parallel_test.py contains """ from mpi4py import MPI import yt yt.enable_parallelism()
ds = yt.load("/scratch/00369/tg456484/Paper49d_moresims/ze01_M10_MA1_1024_quan/DD0095/data0095") /DD%04d/data%04d"%(22,22)) p = yt.ProjectionPlot(ds, "x", "density") p.save() """
my output then looks like:
000 yt : [INFO ] 2019-02-22 13:00:41,879 Gathering a field list (this may take a moment.) File "parallel_test.py", line 13, in <module> p = yt.ProjectionPlot(ds, "x", "density") File "/home1/00369/tg456484/local-yt-2019-02-22-py3/yt-conda/lib/python3.7/site-packages/yt/visualizati on/plot_window.py", line 1480, in __init__ max_level=max_level) File "/home1/00369/tg456484/local-yt-2019-02-22-py3/yt-conda/lib/python3.7/site-packages/yt/data_object s/construction_data_containers.py", line 270, in __init__ self.get_data(field) File "/home1/00369/tg456484/local-yt-2019-02-22-py3/yt-conda/lib/python3.7/site-packages/yt/data_object s/construction_data_containers.py", line 334, in get_data self._initialize_chunk(chunk, tree)
File "/home1/00369/tg456484/local-yt-2019-02-22-py3/yt-conda/lib/python3.7/site-packages/yt/geometry/geometry_handler.py", line 271, in cached_func tr = func(self) File "/home1/00369/tg456484/local-yt-2019-02-22-py3/yt-conda/lib/python3.7/site-packages/yt/geometry/geometry_handler.py", line 332, in icoords ci = np.empty((self.data_size, 3), dtype='int64') P006 yt : [ERROR ] 2019-02-22 13:03:36,145 MemoryError: File "parallel_test.py", line 13, in <module> p = yt.ProjectionPlot(ds, "x", "density") File "/home1/00369/tg456484/local-yt-2019-02-22-py3/yt-conda/lib/python3.7/site-packages/yt/visualization/plot_window.py", line 1480, in __init__ max_level=max_level) File "/home1/00369/tg456484/local-yt-2019-02-22-py3/yt-conda/lib/python3.7/site-packages/yt/data_objects/construction_data_containers.py", line 270, in __init__ self.get_data(field) File "/home1/00369/tg456484/local-yt-2019-02-22-py3/yt-conda/lib/python3.7/site-packages/yt/data_objects/construction_data_containers.py", line 334, in get_data self._initialize_chunk(chunk, tree) File "/home1/00369/tg456484/local-yt-2019-02-22-py3/yt-conda/lib/python3.7/site-packages/yt/data_objects/construction_data_containers.py", line 401, in _initialize_chunk icoords = chunk.icoords File "/home1/00369/tg456484/local-yt-2019-02-22-py3/yt-conda/lib/python3.7/site-packages/yt/data_objects/data_containers.py", line 1555, in icoords return self._current_chunk.icoords File "/home1/00369/tg456484/local-yt-2019-02-22-py3/yt-conda/lib/python3.7/site-packages/yt/geometry/geometry_handler.py", line 271, in cached_func tr = func(self) File "/home1/00369/tg456484/local-yt-2019-02-22-py3/yt-conda/lib/python3.7/site-packages/yt/geometry/geometry_handler.py", line 332, in icoords ci = np.empty((self.data_size, 3), dtype='int64') P007 yt : [ERROR ] 2019-02-22 13:03:36,145 MemoryError: P002 yt : [ERROR ] 2019-02-22 13:03:36,200 Error occured on rank 2. P004 yt : [ERROR ] 2019-02-22 13:03:36,200 Error occured on rank 4. P000 yt : [ERROR ] 2019-02-22 13:03:36,200 Error occured on rank 0. P001 yt : [ERROR ] 2019-02-22 13:03:36,200 Error occured on rank 1. P003 yt : [ERROR ] 2019-02-22 13:03:36,200 Error occured on rank 3. P005 yt : [ERROR ] 2019-02-22 13:03:36,200 Error occured on rank 5. P006 yt : [ERROR ] 2019-02-22 13:03:36,200 Error occured on rank 6. P007 yt : [ERROR ] 2019-02-22 13:03:36,200 Error occured on rank 7. application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0 application called MPI_Abort(MPI_COMM_WORLD, 1) - process 1 application called MPI_Abort(MPI_COMM_WORLD, 1) - process 2 application called MPI_Abort(MPI_COMM_WORLD, 1) - process 3 application called MPI_Abort(MPI_COMM_WORLD, 1) - process 5 application called MPI_Abort(MPI_COMM_WORLD, 1) - process 6 application called MPI_Abort(MPI_COMM_WORLD, 1) - process 7 application called MPI_Abort(MPI_COMM_WORLD, 1) - process 4 TACC: MPI job exited with code: 1 TACC: Shutdown complete. Exiting.
-- -- Sent from a computer. _______________________________________________ yt-users mailing list -- yt-users@python.org To unsubscribe send an email to yt-users-leave@python.org
yt-users mailing list -- yt-users@python.org To unsubscribe send an email to yt-users-leave@python.org
-- -- Sent from a computer. _______________________________________________ yt-users mailing list -- yt-users@python.org To unsubscribe send an email to yt-users-leave@python.org
yt-users mailing list -- yt-users@python.org To unsubscribe send an email to yt-users-leave@python.org
yt-users mailing list -- yt-users@python.org To unsubscribe send an email to yt-users-leave@python.org
participants (3)
-
David Collins -
Matthew Turk -
Nathan Goldbaum