Hello all, including Pragnesh, hope your request went through.
I have since finished the troubling 3200 cube dataset, so good news on my
part, but now I'm trying to track down what made it so difficult.
Let me summarize the problem I had, Pragnesh was continuing a discussion we
had off list:
I was analyzing the 3200 cube, and due to peak memory constraints of
Natuilus, I tried to analyze subvolumes 1/64 of the whole volume one at a
time. The behavior we found was that if we do not end the python script
after each subvolume is analyzed with parallelHF() call, and start
analyzing the subsequent 1/64 of the subvolume, the peak memory will go up,
as per the print statements. If we were to restart the script after each
subvolume is analyzed, we can get the peak memory back to a level
consistent with the first subvolume.
Attached is the memory output plotted of parallelHF with 256 cores, so
every 256 (in the x axis) it looks like a step function. Each step
correspond to the peak memory used by parallelHF on a subvolume, and the
numbers just go up, and if we were to end the calculation in the middle and
restart at a particular subvolume, the numbers would drop back down. Since
the numbers from each core varied, we were wondering if this is because the
particle data remained on the core even after the calculation.
hg tip gave:
user: Stephen Skory <s(a)skory.us>
date: Wed Oct 19 13:44:33 2011 -0600
summary: Merging from mainline.
This is the mod Stephen made to YT to help with this problem about a month
ago. It definitely helped with the memory usage, but I just wanted to
narrow this problem down to something more tangible besides "memory leak"
inside YT. Hopefully others who ran into the same problem or are experts
at parallelHF can help us out.
These issues were found before the new optional KDtree was put in by
Stephen, so that is definitely something we can try. If the problem goes
away, then we can probably say that the memory issue is inside the Fortran
KDtree. I might be the only one encountering this problem, which only
occurs when using YT in unconventional way (I believe the subvolume feature
was made for analyzing a single halo, not the whole volume in pieces), so I
was wondering if anyone else has come across this phenomenon, maybe running
parallelHF on several datasets and seeing a peak memory increase?
On Thu, Nov 17, 2011 at 9:49 AM, Patel, Pragneshkumar B <pragnesh(a)utk.edu>wrote:
*From:* Patel, Pragneshkumar B
*Sent:* Thursday, November 17, 2011 12:18 PM
*To:* Britton Smith
*Cc:* Geoffrey So; s(a)skory.us
*Subject:* RE: [yt-dev] Memory leak issue with ParallelHOP code
I have subscribed to both yt-users and yt-dev. It may possible my requests
are pending. As far as dataset is concern, Geoffrey can give you better
explanation about it.
Dataset contains 0-64 regions. I tried to run ParallelHOP with all regions
but because of memory issue, I could not able to complete all in once. Then
I ran it with only 0-3 regions to debug code "parallel_hop_interface.py".
Please also find attached script. Let me know, if you have more
*From:* Britton Smith [brittonsmith(a)gmail.com]
*Sent:* Thursday, November 17, 2011 12:00 PM
*Cc:* Patel, Pragneshkumar B
*Subject:* Re: [yt-dev] Memory leak issue with ParallelHOP code
Hi again Pragnesh,
I mistakenly asked you to post this to the enzo-users mailing when I meant
to say that you should post this to the yt-users mailing list. Sorry about
that, not enough coffee this morning.
On Thu, Nov 17, 2011 at 8:56 AM, Britton Smith <brittonsmith(a)gmail.com>wrote:
> Hi Pragnesh,
> You need to provide us with some more information. What do you mean when
> you say you ran for 0-3 regions? What kind of dataset are you trying to
> analyze? How big is it? Can you also please post the script you are using
> for this?
> Finally, could you please sign up for the enzo-users mailing list and
> post this there? You've posted this to the development list, which isn't
> the best forum for this question. Also, if you're not on the list you
> won't receive any of the responses to your post.
> On Thu, Nov 17, 2011 at 7:29 AM, Patel, Pragneshkumar B <
> pragnesh(a)utk.edu> wrote:
>> Hi Stephen,
>> I found memory leak issue, when I ran Parallel_HOP code. It may not be
>> in python itself but may be in packages(e.g. kdtree etc..), which we call
>> from"parallel_hop_interface.py". Please find attach file, in which you
>> memory keep increase in every region (I ran program with 32 processes for
>> 0-3 regions.)
>> I like to get your comments/suggestions to solve this issue.
>> I am trying to find more about it using Valgrind. I will let you know.
>> yt-dev mailing list
yt-dev mailing list