Hi Dave,
On Wed, Mar 14, 2012 at 2:18 PM, Matthew Turk
On Wed, Mar 14, 2012 at 12:14 PM, david collins
wrote: 512^3, bunch of subgrids, with 64 cores totalling 12 gb (i.e., just shy of 200mb / core) sounds okay to me. I would guess you are
Ok, I've pulled together some more details.
If I run this 8 cores running a [Density,Density] profile (so, should only have a single field) it fails claiming I'm using 31 Gb, which I have a hard time understanding-- that's nearly 4 Gb per core.
Yeah, puzzling.
I did the same analysis with a more recent (PRGIO) 512^3 with thousands of grids (4 levels by 2) (many more than in the first set), same job script, and it runs fine.
Anyhow, this isn't a huge deal, since non-prgio datasets are far from the norm these days, and it works fine in serial.
Does cProfile work in parallel, and does it do memory work?
You have to manually call it to get it not to overwrite its profile files; I've meant to add something to yt to make this easier, but have not yet. It doesn't do memory work.
For what it's worth, in the PR I just issued for including beta-status Rockstar interfacing, I've added a parallel profiler. You use it like: with parallel_profile("some_prefix"): do_something() something_else() and it will write out .cprof files named as per processor in the top communicator. -Matt
-Matt
d.
seriously disk limited for the analysis itself, but you can verify this by running:
python2.7 -m cProfile -o ytprof.cprof my_script.py
and then using either pyprof2html (requires Jinja2, sold separately) or the stats module to examine it.
-Matt
On Wed, Mar 14, 2012 at 11:15 AM, david collins
wrote: 64 cores, using 12GB on each? That's a problem. MassFraction it
12Gb total, but still... I'll grab some more failure info and get back to you. In serial it ran fine, and took about 1 hour for both [density,cellmass] and [density,cellvolume] within the default head-node allowance on nautilus, which is 12Gb. Timing wise longer than I want to deal with, but it does run.
d.
sounds like has a handful of dependencies -- CellMassMsun, which depends on Density and dx, but dx should be used as a single value not as a field. So that gives 3*512^3*64 bits = 3 gigabytes, but that's a far cry from 12 gb/core. Plus, the domain decomp for profiles is per-grid -- so only one of your cores should get assigned the root grid. This is very bizarre.
Would lazy_reader off change things?
Yes, but it would make them worse. :)
-Matt
Thanks, d.
> Are you running in parallel? Which fields are you profiling? Do they have > lots of dependencies, or require ghost zones? Is yt using lazy_reader? Does > your data have many grids?
> > Matt > > On Mar 14, 2012 12:37 AM, "david collins"
wrote: >> >> I should add that this was done on 64 cores-- in serial it works fine, >> just slow. >> >> On Tue, Mar 13, 2012 at 9:35 PM, david collins >> wrote: >> > Hi, all-- >> > >> > I have an old dataset that I'm trying to make profiles on. It's a >> > 512^3 root grid, but was written with ParallelRootGridIO off. I find >> > that it's using strange amounts of memory, more than 12 Gb. Is this a >> > known problem with a straight forward work-around? >> > >> > d. >> > >> > -- >> > Sent from my computer. >> >> >> >> -- >> Sent from my computer. >> _______________________________________________ >> yt-dev mailing list >> yt-dev@lists.spacepope.org >> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org > > > _______________________________________________ > yt-dev mailing list > yt-dev@lists.spacepope.org > http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org > -- Sent from my computer. _______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
-- Sent from my computer. _______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
-- Sent from my computer. _______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org