Re: [yt-dev] Parallelization & the ART frontend

Hi Chris, This does help. My suspicion is that the load balancing is giving all the lower-level grids to one processor, and all the upper level grids to another. If you have time tomorrow, let's work through this together on IRC. I'm free 1-2PM EST, 3-5PM EST. I think it should be a pretty straightforward fix. It would also be cool to see your new method of accumulating octs into patches. -Matt On Wed, Feb 8, 2012 at 4:09 PM, Christopher Moody wrote:

Hi Matt,

pf.h.proj is of type and refine_by is 2.

Does this help? I'm not sure what you mean by overlaps -  doesn't the the RAMSES grid patching mechanism produce non-overlapping grids from the octs? Is quadtree proj checking for overlapping grids?

chris

On Wed, Feb 8, 2012 at 12:44 PM, Matthew Turk wrote:

...

Hi Chris,

Yeah, that's weird.  My guess is that load balancing is going haywire for some reason, likely due to overlap versus quadtree proj.  Can you tell me what type of object pf.h.proj is?  i.e., what's the output of "print pf.h.proj"?  And then, what's pf.refine_by?

-Matt

On Wed, Feb 8, 2012 at 3:00 PM, Christopher Moody wrote:

...

Hi Matt,

I've got the log output here:http://paste.yt-project.org/show/2153/ with the serial version here http://paste.yt-project.org/show/2154/ .

The most interesting tidbit is below, where it looks like core 0 projects Levels 0-5 and core 1 projects Level 6 (which takes up like 99% of the projection time.)

chris

P001 yt : [DEBUG    ] 2012-02-08 11:39:53,403 Going to obtain [] P001 yt : [DEBUG    ] 2012-02-08 11:39:53,406 Preloading ['density'] from 0 grids P001 yt : [DEBUG    ] 2012-02-08 11:39:53,406 End of projecting level level 0, memory usage 3.545e-01 P001 yt : [DEBUG    ] 2012-02-08 11:39:53,406 Preloading ['density'] from 0 grids P001 yt : [DEBUG    ] 2012-02-08 11:39:53,406 End of projecting level level 1, memory usage 3.545e-01 P001 yt : [DEBUG    ] 2012-02-08 11:39:53,407 Preloading ['density'] from 0 grids P001 yt : [DEBUG    ] 2012-02-08 11:39:53,407 End of projecting level level 2, memory usage 3.545e-01 P001 yt : [DEBUG    ] 2012-02-08 11:39:53,408 Preloading ['density'] from 0 grids P001 yt : [DEBUG    ] 2012-02-08 11:39:53,408 End of projecting level level 3, memory usage 3.545e-01 P001 yt : [DEBUG    ] 2012-02-08 11:39:53,408 Preloading ['density'] from 0 grids P001 yt : [DEBUG    ] 2012-02-08 11:39:53,408 End of projecting level level 4, memory usage 3.545e-01 P001 yt : [DEBUG    ] 2012-02-08 11:39:53,409 Preloading ['density'] from 0 grids P001 yt : [DEBUG    ] 2012-02-08 11:39:53,409 End of projecting level level 5, memory usage 3.545e-01 P001 yt : [DEBUG    ] 2012-02-08 11:39:53,409 Preloading ['density'] from 6 grids P001 yt : [INFO     ] 2012-02-08 11:39:53,410 Starting 'Projecting  level  6 /  6 ' P000 yt : [INFO     ] 2012-02-08 11:39:54,057 Finishing 'Projecting  level  0 /  6 ' P000 yt : [DEBUG    ] 2012-02-08 11:39:54,057 End of projecting level level 0, memory usage 4.482e-01 P000 yt : [DEBUG    ] 2012-02-08 11:39:54,057 Preloading ['density'] from 1 grids P000 yt : [INFO     ] 2012-02-08 11:39:54,058 Starting 'Projecting  level  1 /  6 'P000 yt : [INFO     ] 2012-02-08 11:39:54,070 Finishing 'Projecting  level  1 /  6 ' P000 yt : [DEBUG    ] 2012-02-08 11:39:54,070 End of projecting level level 1, memory usage 4.482e-01 P000 yt : [DEBUG    ] 2012-02-08 11:39:54,070 Preloading ['density'] from 1 gridsP000 yt : [INFO     ] 2012-02-08 11:39:54,071 Starting 'Projecting  level  2 /  6 ' P000 yt : [INFO     ] 2012-02-08 11:39:54,130 Finishing 'Projecting  level  2 /  6 ' P000 yt : [DEBUG    ] 2012-02-08 11:39:54,130 End of projecting level level 2, memory usage 4.482e-01 P000 yt : [DEBUG    ] 2012-02-08 11:39:54,130 Preloading ['density'] from 1 grids P000 yt : [INFO     ] 2012-02-08 11:39:54,131 Starting 'Projecting  level  3 /  6 ' P000 yt : [INFO     ] 2012-02-08 11:39:54,783 Finishing 'Projecting  level  3 /  6 ' P000 yt : [DEBUG    ] 2012-02-08 11:39:54,784 End of projecting level level 3, memory usage 4.482e-01 P000 yt : [DEBUG    ] 2012-02-08 11:39:54,784 Preloading ['density'] from 1 grids P000 yt : [INFO     ] 2012-02-08 11:39:54,784 Starting 'Projecting  level  4 /  6 ' P000 yt : [INFO     ] 2012-02-08 11:39:59,389 Finishing 'Projecting  level  4 /  6 ' P000 yt : [DEBUG    ] 2012-02-08 11:39:59,389 End of projecting level level 4, memory usage 5.918e-01 P000 yt : [DEBUG    ] 2012-02-08 11:39:59,389 Preloading ['density'] from 1 grids P000 yt : [INFO     ] 2012-02-08 11:39:59,389 Starting 'Projecting  level  5 /  6 ' P000 yt : [INFO     ] 2012-02-08 11:40:17,735 Finishing 'Projecting  level  5 /  6 ' P000 yt : [DEBUG    ] 2012-02-08 11:40:17,735 End of projecting level level 5, memory usage 1.569e+00 P000 yt : [DEBUG    ] 2012-02-08 11:40:17,735 Preloading ['density'] from 0 grids P000 yt : [DEBUG    ] 2012-02-08 11:40:17,736 End of projecting level level 6, memory usage 1.569e+00 P001 yt : [INFO     ] 2012-02-08 11:41:31,681 Finishing 'Projecting  level  6 /  6 ' P001 yt : [DEBUG    ] 2012-02-08 11:41:31,681 End of projecting level level 6, memory usage 2.113e+00 P000 yt : [DEBUG    ] 2012-02-08 11:41:33,807 Opening MPI Barrier on 0 P000 yt : [INFO     ] 2012-02-08 11:41:34,502 Projection completed P001 yt : [DEBUG    ] 2012-02-08 11:41:34,502 Opening MPI Barrier on 1 P001 yt : [INFO     ] 2012-02-08 11:41:34,502 Projection completed P001 yt : [DEBUG    ] 2012-02-08 11:41:34,579 Opening MPI Barrier on 1

On Wed, Feb 8, 2012 at 6:46 AM, Matthew Turk wrote:

...

Hi Chris,

On Tue, Feb 7, 2012 at 9:30 PM, Christopher Moody wrote:

...

Hi guys,

I've been working hard on the ART frontend. Lately, I'm to the point where I'm playing around with more complex datasets that are taking much longer to project - so I'd really like to to start using the parallelization engines. I've tried Sam's workshop parallelization demos, and they all work. But launching with the ART frontend (http://paste.yt-project.org/show/2152/) spawns many independent processes which evidently are not actually splitting the projection job, but still taking up lots of processors.

My guess is that parallelism is not enabled for the ART frontend simply as a matter of how the IO is conducted.  To make it really work in parallel, the IO needs to be split up so that when process 1 reads a given grid patch, the rest of the processors don't also need to read all the data for that grid patch.

Can you lower your loglevel (by setting loglevel = 1 in ~/.yt/config or by --config yt.loglevel=1 on the command line) and report back with what it says during a projection job there?

-MAtt

...

My mpi installation  works: yt : [INFO     ] 2012-02-07 18:12:28,207 Global parallel computation enabled: 0 / 8yt : [INFO     ] 2012-02-07 18:12:28,207 Global parallel computation enabled: 2 / 8 yt : [INFO     ] 2012-02-07 18:12:28,208 Global parallel computation enabled: 1 / 8 yt : [INFO     ] 2012-02-07 18:12:28,208 Global parallel computation enabled: 6 / 8 yt : [INFO     ] 2012-02-07 18:12:28,208 Global parallel computation enabled: 3 / 8 yt : [INFO     ] 2012-02-07 18:12:28,208 Global parallel computation enabled: 4 / 8 yt : [INFO     ] 2012-02-07 18:12:28,208 Global parallel computation enabled: 5 / 8yt : [INFO     ] 2012-02-07 18:12:28,209 Global parallel computation enabled: 7 / 8

But the script is just run 8 times, not any faster.

What am I missing here?

Many thanks! chris

_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org

_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org

_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org

_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org

_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org