Sorry to butt in here, but could this be related to the same issue I had when trying to iterate many different subvolume for the halo profiler in a loop style, and see the peak max memory grow over each iteration of the loop? The behavior described seems awfully familiar to me.
The way I got around this problem is to star a new python instance for each of the subvolume in bash, and pass in the subvolume number via sys.argv[].
in bash I have:
for j in {50..50}
do
mpirun -n 256 python <script name>.py $j --parallel 2>&1 | tee `printf %i $j`progress.txt
done
the above will analyze the piece number 50 of my data
in the yt script I have:
import sys
inputsv = int(sys.argv[1])
But of course this is a temporary fix of the symptoms not the cure, since I wasn't able to narrow the problem down further.
From
G.S.
On Sun, Jan 29, 2012 at 9:58 AM, John ZuHone
<jzuhone@milkyway.gsfc.nasa.gov> wrote:
Thanks to everyone for the help. The files are not very big, so I think this is an issue of memory not being freed up as it should, as per Matt's suggestion. I'm rewriting the script as he suggested and then trying that.
John
On Jan 29, 2012, at 2:13 AM, Sam Skillman wrote:
Hi John,
Additionally, adding 'del cam, image' at the end of each loop should help considerably. The camera object contains the homogenized volume roughly equal to the size of the entire dataset for one field, so I'm guessing that is where most of it is coming from.
Let us know if that doesn't help.
Sam
On Sat, Jan 28, 2012 at 7:31 PM, Nathan Goldbaum
<goldbaum@ucolick.org> wrote:
Hi John,
I've found that calling the garbage collector via gc.collect() at the end of a loop over filenames fixes issues like this. Probably worth a try, anyway.
-Nathan
> !DSPAM:10175,4f24aeed3642254826273!
yt-users mailing list
yt-users@lists.spacepope.org
http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________
yt-users mailing list
yt-users@lists.spacepope.org
http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org