Hi, I have pushed my changes into a new bugfix branch: https://github.com/yt-project/yt/pull/4164. Don't hesitate to chime in if you want to add extra information or simply to confirm it is indeed solving the memory leak! Best, Corentin On 12/10/2022 21:10, Anne Noer Kolborg wrote:
Hi all,
Thank you very much for your help on this! I appreciate all your inputs.
From preliminary testing it looks like the patch from Corentin has fixed the issue.
Let me know if it would be helpful that I open an issue about this on GitHub.
Best regards, Anne
On 12 Oct 2022, at 00.29, Corentin CADIOU <contact@cphyc.me> wrote:
Hi all,
Would it be possible to test the following patch and tell me whether that solves the memory issue? I was unable to reproduce the memory leak on my machine, but I have terrible internet access at the moment and I only have smallish datasets locally.
diff --git a/yt/geometry/oct_container.pyx b/yt/geometry/oct_container.pyx index b0cbd6a433..58c31ce5d2 100644 --- a/yt/geometry/oct_container.pyx +++ b/yt/geometry/oct_container.pyx @@ -1073,6 +1073,11 @@ cdef class SparseOctreeContainer(OctreeContainer): def __dealloc__(self): # This gets called BEFORE the superclass deallocation. But, both get # called. + cdef OctKey *ikey + for i in range(self.num_root): + ikey = &self.root_nodes[i] + tdelete(<void *>ikey, &self.tree_root, root_node_compare) + if self.root_nodes != NULL: free(self.root_nodes)
cdef class ARTOctreeContainer(OctreeContainer):
What I think is happening is that|tsearch|, which was pointed out by Shin-Rong as the main culprit, is doing a lot of|malloc|but the memory is never freed again. This will lead to allocating|2**levelmin|for RAMSES chunks of data being allocated. The function is used to handle the AMR grid.
Best, Corentin
On 11/10/2022 19:57, Jacob Morgan wrote:
I believe a similar issue exists in the AREPO frontend (maybe if it exists for both the frontend is not the problem...?). Sorry I don't have more details, someone in my research group was complaining of a memory leak when loading AREPO datasets one after another. ------------------------------------------------------------------------ *From:*Clément Robert via yt-users<yt-users@python.org> *Sent:*Tuesday, October 11, 2022 9:51 AM *To:*Discussion of the yt analysis package<yt-users@python.org> *Cc:*Anne Noer Kolborg<akolborg@ucsc.edu>; Nicholas 💍🐻❤️<nicholas@swiatecki.com>; Clément Robert<clement.robert@protonmail.com> *Subject:*[EXTERNAL] [yt-users] Re: Potential memory leak in Ramses frontend As a quick pointer, I’d like to promote Memray, which is still relatively new, and an excellent tool to trace an visualise memory usage in Python applications in details https://github.com/bloomberg/memray <https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fbloomberg%2Fmemray&data=05%7C01%7Cjmorgan15%40crimson.ua.edu%7Cd0dae389465a4f20c3da08daab9838e5%7C2a00728ef0d040b4a4e8ce433f3fbca7%7C0%7C0%7C638010967575578774%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=lEW1ymCzHMuvTHoFMplFKZ4sQZNuy%2FDJsBjyXZV7LwE%3D&reserved=0>
cheers Clément
On 11 Oct 2022, at 05:48, Anne Noer Kolborg via yt-users <yt-users@python.org> wrote:
Dear yt community,
I am have encountered an out-of-memory problem (potential memory leak) when using yt to load and process several Ramses outputs in a row. Each simulation output is approximately 3 GB, but the RAM usage easily exceeds 16 GB in ~30 mins of computation time (single threaded computation time). When allocated more RAM the computation continues for longer but ultimately always runs out the memory before the end of the data-processing.
I have created a minimal working example that triggers this problem. It is uploaded here:http://paste.yt-project.org/show/404/ <https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Fpaste.yt-project.org%2Fshow%2F404%2F&data=05%7C01%7Cjmorgan15%40crimson.ua.edu%7Cd0dae389465a4f20c3da08daab9838e5%7C2a00728ef0d040b4a4e8ce433f3fbca7%7C0%7C0%7C638010967575578774%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=LOQzTjGIcN5eEDqCOEswtvVmMjpK%2FopplosYkOiGQ%2Fo%3D&reserved=0>
Log of the memory usage over time while running that script is here:http://paste.yt-project.org/show/406/ <https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Fpaste.yt-project.org%2Fshow%2F405%2F&data=05%7C01%7Cjmorgan15%40crimson.ua.edu%7Cd0dae389465a4f20c3da08daab9838e5%7C2a00728ef0d040b4a4e8ce433f3fbca7%7C0%7C0%7C638010967575578774%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=if%2Fm4vJC5RHVAD4NmUCIxPD2nQ6Gan0LTb8%2FmvNS0oA%3D&reserved=0>
As you can see the total memory usage simply grows for each round of the loop. I suspect that the memory allocated to the dataset does not get cleared.
Simulation output to test the code is uploaded here:http://use.yt/upload/1ace6072 <https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Fuse.yt%2Fupload%2F1ace6072&data=05%7C01%7Cjmorgan15%40crimson.ua.edu%7Cd0dae389465a4f20c3da08daab9838e5%7C2a00728ef0d040b4a4e8ce433f3fbca7%7C0%7C0%7C638010967575891276%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=YVUUX4ANsy2WoMcnY5OLEUxJfxSg9lMSQM2sbtC%2Bxf8%3D&reserved=0>
The del and clear_data() statements in line 18 and 19 do not impact the memory usage when commented in.
I tested the same bit of code using the “HiresIsolatedGalaxy” Enzo dataset from the sample datasets (https://yt-project.org/data/ <https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fyt-project.org%2Fdata%2F&data=05%7C01%7Cjmorgan15%40crimson.ua.edu%7Cd0dae389465a4f20c3da08daab9838e5%7C2a00728ef0d040b4a4e8ce433f3fbca7%7C0%7C0%7C638010967575891276%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=24oomFsnD2dZj0ZCVHpqVbzZ9RZfTO458koQ05PqgFQ%3D&reserved=0>). It is similar in size to the outputs I am trying to process and runs without issues. Therefore, I suspect it might be the ramses frontend causing the memory leak.
Any advice or pointers on how to debug this further, alternative ways to handle this or suggestions for how to use yt more efficiently are most welcome!
Best regards, Anne
_ _ _Version lists: _ Yt version 4.1.dev, changeset = 39949fb6bfba. (cloned from GitHub about a week ago) - the stable version 4.0.1 also produces the same issue. Python version 3.8.12 Gcc 4.8.5 CentOS Linux release 7.9.2009 // Linux 3.10.0 (x86-64 architecture)
_______________________________________________ yt-users mailing list --yt-users@python.org To unsubscribe send an email toyt-users-leave@python.org https://mail.python.org/mailman3/lists/yt-users.python.org/ Member address:contact@cphyc.me
<OpenPGP_0xEEBF77E6D1B29577.asc>