Potential memory leak in Ramses frontend
Dear yt community, I am have encountered an out-of-memory problem (potential memory leak) when using yt to load and process several Ramses outputs in a row. Each simulation output is approximately 3 GB, but the RAM usage easily exceeds 16 GB in ~30 mins of computation time (single threaded computation time). When allocated more RAM the computation continues for longer but ultimately always runs out the memory before the end of the data-processing. I have created a minimal working example that triggers this problem. It is uploaded here: http://paste.yt-project.org/show/404/ <http://paste.yt-project.org/show/404/> Log of the memory usage over time while running that script is here: http://paste.yt-project.org/show/406/ <http://paste.yt-project.org/show/405/> As you can see the total memory usage simply grows for each round of the loop. I suspect that the memory allocated to the dataset does not get cleared. Simulation output to test the code is uploaded here: http://use.yt/upload/1ace6072 <http://use.yt/upload/1ace6072> The del and clear_data() statements in line 18 and 19 do not impact the memory usage when commented in. I tested the same bit of code using the “HiresIsolatedGalaxy” Enzo dataset from the sample datasets (https://yt-project.org/data/ <https://yt-project.org/data/>). It is similar in size to the outputs I am trying to process and runs without issues. Therefore, I suspect it might be the ramses frontend causing the memory leak. Any advice or pointers on how to debug this further, alternative ways to handle this or suggestions for how to use yt more efficiently are most welcome! Best regards, Anne Version lists: Yt version 4.1.dev, changeset = 39949fb6bfba. (cloned from GitHub about a week ago) - the stable version 4.0.1 also produces the same issue. Python version 3.8.12 Gcc 4.8.5 CentOS Linux release 7.9.2009 // Linux 3.10.0 (x86-64 architecture)
Dear Anne, Many thanks for the detailed report. It may very well be there is a memory leak somewhere in the code. Could you however try calling |import gc; gc.collect() | after calling |del| and / or |clear_data()|? This should force garbage-collection. Best, Corentin On 11/10/2022 05:48, Anne Noer Kolborg via yt-users wrote:
Dear yt community,
I am have encountered an out-of-memory problem (potential memory leak) when using yt to load and process several Ramses outputs in a row. Each simulation output is approximately 3 GB, but the RAM usage easily exceeds 16 GB in ~30 mins of computation time (single threaded computation time). When allocated more RAM the computation continues for longer but ultimately always runs out the memory before the end of the data-processing.
I have created a minimal working example that triggers this problem. It is uploaded here: http://paste.yt-project.org/show/404/ Log of the memory usage over time while running that script is here: http://paste.yt-project.org/show/406/ <http://paste.yt-project.org/show/405/>
As you can see the total memory usage simply grows for each round of the loop. I suspect that the memory allocated to the dataset does not get cleared.
Simulation output to test the code is uploaded here: http://use.yt/upload/1ace6072
The del and clear_data() statements in line 18 and 19 do not impact the memory usage when commented in.
I tested the same bit of code using the “HiresIsolatedGalaxy” Enzo dataset from the sample datasets (https://yt-project.org/data/). It is similar in size to the outputs I am trying to process and runs without issues. Therefore, I suspect it might be the ramses frontend causing the memory leak.
Any advice or pointers on how to debug this further, alternative ways to handle this or suggestions for how to use yt more efficiently are most welcome!
Best regards, Anne
_ _ _Version lists: _ Yt version 4.1.dev, changeset = 39949fb6bfba. (cloned from GitHub about a week ago) - the stable version 4.0.1 also produces the same issue. Python version 3.8.12 Gcc 4.8.5 CentOS Linux release 7.9.2009 // Linux 3.10.0 (x86-64 architecture)
_______________________________________________ yt-users mailing list --yt-users@python.org To unsubscribe send an email toyt-users-leave@python.org https://mail.python.org/mailman3/lists/yt-users.python.org/ Member address:contact@cphyc.me
As a quick pointer, I’d like to promote Memray, which is still relatively new, and an excellent tool to trace an visualise memory usage in Python applications in detailshttps://github.com/bloomberg/memray cheers Clément
On 11 Oct 2022, at 05:48, Anne Noer Kolborg via yt-users <yt-users@python.org> wrote:
Dear yt community,
I am have encountered an out-of-memory problem (potential memory leak) when using yt to load and process several Ramses outputs in a row. Each simulation output is approximately 3 GB, but the RAM usage easily exceeds 16 GB in ~30 mins of computation time (single threaded computation time). When allocated more RAM the computation continues for longer but ultimately always runs out the memory before the end of the data-processing.
I have created a minimal working example that triggers this problem. It is uploaded here: http://paste.yt-project.org/show/404/ Log of the memory usage over time while running that script is here: [http://paste.yt-project.org/show/406/](http://paste.yt-project.org/show/405/)
As you can see the total memory usage simply grows for each round of the loop. I suspect that the memory allocated to the dataset does not get cleared.
Simulation output to test the code is uploaded here: http://use.yt/upload/1ace6072
The del and clear_data() statements in line 18 and 19 do not impact the memory usage when commented in.
I tested the same bit of code using the “HiresIsolatedGalaxy” Enzo dataset from the sample datasets (https://yt-project.org/data/). It is similar in size to the outputs I am trying to process and runs without issues. Therefore, I suspect it might be the ramses frontend causing the memory leak.
Any advice or pointers on how to debug this further, alternative ways to handle this or suggestions for how to use yt more efficiently are most welcome!
Best regards, Anne
Version lists: Yt version 4.1.dev, changeset = 39949fb6bfba. (cloned from GitHub about a week ago) - the stable version 4.0.1 also produces the same issue. Python version 3.8.12 Gcc 4.8.5 CentOS Linux release 7.9.2009 // Linux 3.10.0 (x86-64 architecture)
I believe a similar issue exists in the AREPO frontend (maybe if it exists for both the frontend is not the problem...?). Sorry I don't have more details, someone in my research group was complaining of a memory leak when loading AREPO datasets one after another. ________________________________ From: Clément Robert via yt-users <yt-users@python.org> Sent: Tuesday, October 11, 2022 9:51 AM To: Discussion of the yt analysis package <yt-users@python.org> Cc: Anne Noer Kolborg <akolborg@ucsc.edu>; Nicholas 💍🐻❤️ <nicholas@swiatecki.com>; Clément Robert <clement.robert@protonmail.com> Subject: [EXTERNAL] [yt-users] Re: Potential memory leak in Ramses frontend As a quick pointer, I’d like to promote Memray, which is still relatively new, and an excellent tool to trace an visualise memory usage in Python applications in details https://github.com/bloomberg/memray<https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fbloomberg%2Fmemray&data=05%7C01%7Cjmorgan15%40crimson.ua.edu%7Cd0dae389465a4f20c3da08daab9838e5%7C2a00728ef0d040b4a4e8ce433f3fbca7%7C0%7C0%7C638010967575578774%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=lEW1ymCzHMuvTHoFMplFKZ4sQZNuy%2FDJsBjyXZV7LwE%3D&reserved=0> cheers Clément On 11 Oct 2022, at 05:48, Anne Noer Kolborg via yt-users <yt-users@python.org<mailto:yt-users@python.org>> wrote: Dear yt community, I am have encountered an out-of-memory problem (potential memory leak) when using yt to load and process several Ramses outputs in a row. Each simulation output is approximately 3 GB, but the RAM usage easily exceeds 16 GB in ~30 mins of computation time (single threaded computation time). When allocated more RAM the computation continues for longer but ultimately always runs out the memory before the end of the data-processing. I have created a minimal working example that triggers this problem. It is uploaded here: http://paste.yt-project.org/show/404/<https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Fpaste.yt-project.org%2Fshow%2F404%2F&data=05%7C01%7Cjmorgan15%40crimson.ua.edu%7Cd0dae389465a4f20c3da08daab9838e5%7C2a00728ef0d040b4a4e8ce433f3fbca7%7C0%7C0%7C638010967575578774%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=LOQzTjGIcN5eEDqCOEswtvVmMjpK%2FopplosYkOiGQ%2Fo%3D&reserved=0> Log of the memory usage over time while running that script is here: http://paste.yt-project.org/show/406/<https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Fpaste.yt-project.org%2Fshow%2F405%2F&data=05%7C01%7Cjmorgan15%40crimson.ua.edu%7Cd0dae389465a4f20c3da08daab9838e5%7C2a00728ef0d040b4a4e8ce433f3fbca7%7C0%7C0%7C638010967575578774%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=if%2Fm4vJC5RHVAD4NmUCIxPD2nQ6Gan0LTb8%2FmvNS0oA%3D&reserved=0> As you can see the total memory usage simply grows for each round of the loop. I suspect that the memory allocated to the dataset does not get cleared. Simulation output to test the code is uploaded here: http://use.yt/upload/1ace6072<https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Fuse.yt%2Fupload%2F1ace6072&data=05%7C01%7Cjmorgan15%40crimson.ua.edu%7Cd0dae389465a4f20c3da08daab9838e5%7C2a00728ef0d040b4a4e8ce433f3fbca7%7C0%7C0%7C638010967575891276%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=YVUUX4ANsy2WoMcnY5OLEUxJfxSg9lMSQM2sbtC%2Bxf8%3D&reserved=0> The del and clear_data() statements in line 18 and 19 do not impact the memory usage when commented in. I tested the same bit of code using the “HiresIsolatedGalaxy” Enzo dataset from the sample datasets (https://yt-project.org/data/<https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fyt-project.org%2Fdata%2F&data=05%7C01%7Cjmorgan15%40crimson.ua.edu%7Cd0dae389465a4f20c3da08daab9838e5%7C2a00728ef0d040b4a4e8ce433f3fbca7%7C0%7C0%7C638010967575891276%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=24oomFsnD2dZj0ZCVHpqVbzZ9RZfTO458koQ05PqgFQ%3D&reserved=0>). It is similar in size to the outputs I am trying to process and runs without issues. Therefore, I suspect it might be the ramses frontend causing the memory leak. Any advice or pointers on how to debug this further, alternative ways to handle this or suggestions for how to use yt more efficiently are most welcome! Best regards, Anne Version lists: Yt version 4.1.dev, changeset = 39949fb6bfba. (cloned from GitHub about a week ago) - the stable version 4.0.1 also produces the same issue. Python version 3.8.12 Gcc 4.8.5 CentOS Linux release 7.9.2009 // Linux 3.10.0 (x86-64 architecture)
Dear Anne, I ran some memory checking test using dataset `HiresIsolatedGalaxy` (Enzo), `JetICMWall` (Gamer), and yours (Ramses) with the same Python script you posted. It looks like memory for tsearch in module libc is piling up. Please see the results here (https://hackmd.io/@Viukb0eMS-aeoZQudVyJ2w/SkIcZDmXo#Use-Valgrind-to-Check-Me...). Also, you may ignore the Related Issue section. Best regards, Shin-Rong Tsai Anne Noer Kolborg wrote:
Dear yt community, I am have encountered an out-of-memory problem (potential memory leak) when using yt to load and process several Ramses outputs in a row. Each simulation output is approximately 3 GB, but the RAM usage easily exceeds 16 GB in ~30 mins of computation time (single threaded computation time). When allocated more RAM the computation continues for longer but ultimately always runs out the memory before the end of the data-processing. I have created a minimal working example that triggers this problem. It is uploaded here: http://paste.yt-project.org/show/404/ http://paste.yt-project.org/show/404/ Log of the memory usage over time while running that script is here: http://paste.yt-project.org/show/406/ http://paste.yt-project.org/show/405/ As you can see the total memory usage simply grows for each round of the loop. I suspect that the memory allocated to the dataset does not get cleared. Simulation output to test the code is uploaded here: http://use.yt/upload/1ace6072 http://use.yt/upload/1ace6072 The del and clear_data() statements in line 18 and 19 do not impact the memory usage when commented in. I tested the same bit of code using the “HiresIsolatedGalaxy” Enzo dataset from the sample datasets (https://yt-project.org/data/ https://yt-project.org/data/). It is similar in size to the outputs I am trying to process and runs without issues. Therefore, I suspect it might be the ramses frontend causing the memory leak. Any advice or pointers on how to debug this further, alternative ways to handle this or suggestions for how to use yt more efficiently are most welcome! Best regards, Anne Version lists: Yt version 4.1.dev, changeset = 39949fb6bfba. (cloned from GitHub about a week ago) - the stable version 4.0.1 also produces the same issue. Python version 3.8.12 Gcc 4.8.5 CentOS Linux release 7.9.2009 // Linux 3.10.0 (x86-64 architecture)
Hi all, Would it be possible to test the following patch and tell me whether that solves the memory issue? I was unable to reproduce the memory leak on my machine, but I have terrible internet access at the moment and I only have smallish datasets locally. |diff --git a/yt/geometry/oct_container.pyx b/yt/geometry/oct_container.pyx index b0cbd6a433..58c31ce5d2 100644 --- a/yt/geometry/oct_container.pyx +++ b/yt/geometry/oct_container.pyx @@ -1073,6 +1073,11 @@ cdef class SparseOctreeContainer(OctreeContainer): def __dealloc__(self): # This gets called BEFORE the superclass deallocation. But, both get # called. + cdef OctKey *ikey + for i in range(self.num_root): + ikey = &self.root_nodes[i] + tdelete(<void *>ikey, &self.tree_root, root_node_compare) + if self.root_nodes != NULL: free(self.root_nodes) cdef class ARTOctreeContainer(OctreeContainer): | What I think is happening is that |tsearch|, which was pointed out by Shin-Rong as the main culprit, is doing a lot of |malloc| but the memory is never freed again. This will lead to allocating |2**levelmin| for RAMSES chunks of data being allocated. The function is used to handle the AMR grid. Best, Corentin On 11/10/2022 19:57, Jacob Morgan wrote:
I believe a similar issue exists in the AREPO frontend (maybe if it exists for both the frontend is not the problem...?). Sorry I don't have more details, someone in my research group was complaining of a memory leak when loading AREPO datasets one after another. ------------------------------------------------------------------------ *From:* Clément Robert via yt-users <yt-users@python.org> *Sent:* Tuesday, October 11, 2022 9:51 AM *To:* Discussion of the yt analysis package <yt-users@python.org> *Cc:* Anne Noer Kolborg <akolborg@ucsc.edu>; Nicholas 💍🐻❤️ <nicholas@swiatecki.com>; Clément Robert <clement.robert@protonmail.com> *Subject:* [EXTERNAL] [yt-users] Re: Potential memory leak in Ramses frontend As a quick pointer, I’d like to promote Memray, which is still relatively new, and an excellent tool to trace an visualise memory usage in Python applications in details https://github.com/bloomberg/memray <https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fbloomberg%2Fmemray&data=05%7C01%7Cjmorgan15%40crimson.ua.edu%7Cd0dae389465a4f20c3da08daab9838e5%7C2a00728ef0d040b4a4e8ce433f3fbca7%7C0%7C0%7C638010967575578774%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=lEW1ymCzHMuvTHoFMplFKZ4sQZNuy%2FDJsBjyXZV7LwE%3D&reserved=0>
cheers Clément
On 11 Oct 2022, at 05:48, Anne Noer Kolborg via yt-users <yt-users@python.org> wrote:
Dear yt community,
I am have encountered an out-of-memory problem (potential memory leak) when using yt to load and process several Ramses outputs in a row. Each simulation output is approximately 3 GB, but the RAM usage easily exceeds 16 GB in ~30 mins of computation time (single threaded computation time). When allocated more RAM the computation continues for longer but ultimately always runs out the memory before the end of the data-processing.
I have created a minimal working example that triggers this problem. It is uploaded here: http://paste.yt-project.org/show/404/ <https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Fpaste.yt-project.org%2Fshow%2F404%2F&data=05%7C01%7Cjmorgan15%40crimson.ua.edu%7Cd0dae389465a4f20c3da08daab9838e5%7C2a00728ef0d040b4a4e8ce433f3fbca7%7C0%7C0%7C638010967575578774%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=LOQzTjGIcN5eEDqCOEswtvVmMjpK%2FopplosYkOiGQ%2Fo%3D&reserved=0>
Log of the memory usage over time while running that script is here: http://paste.yt-project.org/show/406/ <https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Fpaste.yt-project.org%2Fshow%2F405%2F&data=05%7C01%7Cjmorgan15%40crimson.ua.edu%7Cd0dae389465a4f20c3da08daab9838e5%7C2a00728ef0d040b4a4e8ce433f3fbca7%7C0%7C0%7C638010967575578774%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=if%2Fm4vJC5RHVAD4NmUCIxPD2nQ6Gan0LTb8%2FmvNS0oA%3D&reserved=0>
As you can see the total memory usage simply grows for each round of the loop. I suspect that the memory allocated to the dataset does not get cleared.
Simulation output to test the code is uploaded here: http://use.yt/upload/1ace6072 <https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Fuse.yt%2Fupload%2F1ace6072&data=05%7C01%7Cjmorgan15%40crimson.ua.edu%7Cd0dae389465a4f20c3da08daab9838e5%7C2a00728ef0d040b4a4e8ce433f3fbca7%7C0%7C0%7C638010967575891276%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=YVUUX4ANsy2WoMcnY5OLEUxJfxSg9lMSQM2sbtC%2Bxf8%3D&reserved=0>
The del and clear_data() statements in line 18 and 19 do not impact the memory usage when commented in.
I tested the same bit of code using the “HiresIsolatedGalaxy” Enzo dataset from the sample datasets (https://yt-project.org/data/ <https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fyt-project.org%2Fdata%2F&data=05%7C01%7Cjmorgan15%40crimson.ua.edu%7Cd0dae389465a4f20c3da08daab9838e5%7C2a00728ef0d040b4a4e8ce433f3fbca7%7C0%7C0%7C638010967575891276%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=24oomFsnD2dZj0ZCVHpqVbzZ9RZfTO458koQ05PqgFQ%3D&reserved=0>). It is similar in size to the outputs I am trying to process and runs without issues. Therefore, I suspect it might be the ramses frontend causing the memory leak.
Any advice or pointers on how to debug this further, alternative ways to handle this or suggestions for how to use yt more efficiently are most welcome!
Best regards, Anne
_ _ _Version lists: _ Yt version 4.1.dev, changeset = 39949fb6bfba. (cloned from GitHub about a week ago) - the stable version 4.0.1 also produces the same issue. Python version 3.8.12 Gcc 4.8.5 CentOS Linux release 7.9.2009 // Linux 3.10.0 (x86-64 architecture)
_______________________________________________ yt-users mailing list --yt-users@python.org To unsubscribe send an email toyt-users-leave@python.org https://mail.python.org/mailman3/lists/yt-users.python.org/ Member address:contact@cphyc.me
Hi all, Thank you very much for your help on this! I appreciate all your inputs. From preliminary testing it looks like the patch from Corentin has fixed the issue. Let me know if it would be helpful that I open an issue about this on GitHub. Best regards, Anne
On 12 Oct 2022, at 00.29, Corentin CADIOU <contact@cphyc.me> wrote:
Hi all,
Would it be possible to test the following patch and tell me whether that solves the memory issue? I was unable to reproduce the memory leak on my machine, but I have terrible internet access at the moment and I only have smallish datasets locally.
diff --git a/yt/geometry/oct_container.pyx b/yt/geometry/oct_container.pyx index b0cbd6a433..58c31ce5d2 100644 --- a/yt/geometry/oct_container.pyx +++ b/yt/geometry/oct_container.pyx @@ -1073,6 +1073,11 @@ cdef class SparseOctreeContainer(OctreeContainer): def __dealloc__(self): # This gets called BEFORE the superclass deallocation. But, both get # called. + cdef OctKey *ikey + for i in range(self.num_root): + ikey = &self.root_nodes[i] + tdelete(<void *>ikey, &self.tree_root, root_node_compare) + if self.root_nodes != NULL: free(self.root_nodes)
cdef class ARTOctreeContainer(OctreeContainer): What I think is happening is that tsearch, which was pointed out by Shin-Rong as the main culprit, is doing a lot of malloc but the memory is never freed again. This will lead to allocating 2**levelmin for RAMSES chunks of data being allocated. The function is used to handle the AMR grid.
Best, Corentin
On 11/10/2022 19:57, Jacob Morgan wrote:
I believe a similar issue exists in the AREPO frontend (maybe if it exists for both the frontend is not the problem...?). Sorry I don't have more details, someone in my research group was complaining of a memory leak when loading AREPO datasets one after another. From: Clément Robert via yt-users <yt-users@python.org> <mailto:yt-users@python.org> Sent: Tuesday, October 11, 2022 9:51 AM To: Discussion of the yt analysis package <yt-users@python.org> <mailto:yt-users@python.org> Cc: Anne Noer Kolborg <akolborg@ucsc.edu> <mailto:akolborg@ucsc.edu>; Nicholas 💍🐻❤️ <nicholas@swiatecki.com> <mailto:nicholas@swiatecki.com>; Clément Robert <clement.robert@protonmail.com> <mailto:clement.robert@protonmail.com> Subject: [EXTERNAL] [yt-users] Re: Potential memory leak in Ramses frontend
As a quick pointer, I’d like to promote Memray, which is still relatively new, and an excellent tool to trace an visualise memory usage in Python applications in details https://github.com/bloomberg/memray <https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fbloomberg%2Fmemray&data=05%7C01%7Cjmorgan15%40crimson.ua.edu%7Cd0dae389465a4f20c3da08daab9838e5%7C2a00728ef0d040b4a4e8ce433f3fbca7%7C0%7C0%7C638010967575578774%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=lEW1ymCzHMuvTHoFMplFKZ4sQZNuy%2FDJsBjyXZV7LwE%3D&reserved=0>
cheers Clément
On 11 Oct 2022, at 05:48, Anne Noer Kolborg via yt-users <yt-users@python.org <mailto:yt-users@python.org>> wrote:
Dear yt community,
I am have encountered an out-of-memory problem (potential memory leak) when using yt to load and process several Ramses outputs in a row. Each simulation output is approximately 3 GB, but the RAM usage easily exceeds 16 GB in ~30 mins of computation time (single threaded computation time). When allocated more RAM the computation continues for longer but ultimately always runs out the memory before the end of the data-processing.
I have created a minimal working example that triggers this problem. It is uploaded here: http://paste.yt-project.org/show/404/ <https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Fpaste.yt-project.org%2Fshow%2F404%2F&data=05%7C01%7Cjmorgan15%40crimson.ua.edu%7Cd0dae389465a4f20c3da08daab9838e5%7C2a00728ef0d040b4a4e8ce433f3fbca7%7C0%7C0%7C638010967575578774%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=LOQzTjGIcN5eEDqCOEswtvVmMjpK%2FopplosYkOiGQ%2Fo%3D&reserved=0> Log of the memory usage over time while running that script is here: http://paste.yt-project.org/show/406/ <https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Fpaste.yt-project.org%2Fshow%2F405%2F&data=05%7C01%7Cjmorgan15%40crimson.ua.edu%7Cd0dae389465a4f20c3da08daab9838e5%7C2a00728ef0d040b4a4e8ce433f3fbca7%7C0%7C0%7C638010967575578774%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=if%2Fm4vJC5RHVAD4NmUCIxPD2nQ6Gan0LTb8%2FmvNS0oA%3D&reserved=0>
As you can see the total memory usage simply grows for each round of the loop. I suspect that the memory allocated to the dataset does not get cleared.
Simulation output to test the code is uploaded here: http://use.yt/upload/1ace6072 <https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Fuse.yt%2Fupload%2F1ace6072&data=05%7C01%7Cjmorgan15%40crimson.ua.edu%7Cd0dae389465a4f20c3da08daab9838e5%7C2a00728ef0d040b4a4e8ce433f3fbca7%7C0%7C0%7C638010967575891276%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=YVUUX4ANsy2WoMcnY5OLEUxJfxSg9lMSQM2sbtC%2Bxf8%3D&reserved=0>
The del and clear_data() statements in line 18 and 19 do not impact the memory usage when commented in.
I tested the same bit of code using the “HiresIsolatedGalaxy” Enzo dataset from the sample datasets (https://yt-project.org/data/ <https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fyt-project.org%2Fdata%2F&data=05%7C01%7Cjmorgan15%40crimson.ua.edu%7Cd0dae389465a4f20c3da08daab9838e5%7C2a00728ef0d040b4a4e8ce433f3fbca7%7C0%7C0%7C638010967575891276%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=24oomFsnD2dZj0ZCVHpqVbzZ9RZfTO458koQ05PqgFQ%3D&reserved=0>). It is similar in size to the outputs I am trying to process and runs without issues. Therefore, I suspect it might be the ramses frontend causing the memory leak.
Any advice or pointers on how to debug this further, alternative ways to handle this or suggestions for how to use yt more efficiently are most welcome!
Best regards, Anne
Version lists: Yt version 4.1.dev, changeset = 39949fb6bfba. (cloned from GitHub about a week ago) - the stable version 4.0.1 also produces the same issue. Python version 3.8.12 Gcc 4.8.5 CentOS Linux release 7.9.2009 // Linux 3.10.0 (x86-64 architecture)
_______________________________________________ yt-users mailing list -- yt-users@python.org <mailto:yt-users@python.org> To unsubscribe send an email to yt-users-leave@python.org <mailto:yt-users-leave@python.org> https://mail.python.org/mailman3/lists/yt-users.python.org/ <https://mail.python.org/mailman3/lists/yt-users.python.org/> Member address: contact@cphyc.me <mailto:contact@cphyc.me>
<OpenPGP_0xEEBF77E6D1B29577.asc>
Hi, I have pushed my changes into a new bugfix branch: https://github.com/yt-project/yt/pull/4164. Don't hesitate to chime in if you want to add extra information or simply to confirm it is indeed solving the memory leak! Best, Corentin On 12/10/2022 21:10, Anne Noer Kolborg wrote:
Hi all,
Thank you very much for your help on this! I appreciate all your inputs.
From preliminary testing it looks like the patch from Corentin has fixed the issue.
Let me know if it would be helpful that I open an issue about this on GitHub.
Best regards, Anne
On 12 Oct 2022, at 00.29, Corentin CADIOU <contact@cphyc.me> wrote:
Hi all,
Would it be possible to test the following patch and tell me whether that solves the memory issue? I was unable to reproduce the memory leak on my machine, but I have terrible internet access at the moment and I only have smallish datasets locally.
diff --git a/yt/geometry/oct_container.pyx b/yt/geometry/oct_container.pyx index b0cbd6a433..58c31ce5d2 100644 --- a/yt/geometry/oct_container.pyx +++ b/yt/geometry/oct_container.pyx @@ -1073,6 +1073,11 @@ cdef class SparseOctreeContainer(OctreeContainer): def __dealloc__(self): # This gets called BEFORE the superclass deallocation. But, both get # called. + cdef OctKey *ikey + for i in range(self.num_root): + ikey = &self.root_nodes[i] + tdelete(<void *>ikey, &self.tree_root, root_node_compare) + if self.root_nodes != NULL: free(self.root_nodes)
cdef class ARTOctreeContainer(OctreeContainer):
What I think is happening is that|tsearch|, which was pointed out by Shin-Rong as the main culprit, is doing a lot of|malloc|but the memory is never freed again. This will lead to allocating|2**levelmin|for RAMSES chunks of data being allocated. The function is used to handle the AMR grid.
Best, Corentin
On 11/10/2022 19:57, Jacob Morgan wrote:
I believe a similar issue exists in the AREPO frontend (maybe if it exists for both the frontend is not the problem...?). Sorry I don't have more details, someone in my research group was complaining of a memory leak when loading AREPO datasets one after another. ------------------------------------------------------------------------ *From:*Clément Robert via yt-users<yt-users@python.org> *Sent:*Tuesday, October 11, 2022 9:51 AM *To:*Discussion of the yt analysis package<yt-users@python.org> *Cc:*Anne Noer Kolborg<akolborg@ucsc.edu>; Nicholas 💍🐻❤️<nicholas@swiatecki.com>; Clément Robert<clement.robert@protonmail.com> *Subject:*[EXTERNAL] [yt-users] Re: Potential memory leak in Ramses frontend As a quick pointer, I’d like to promote Memray, which is still relatively new, and an excellent tool to trace an visualise memory usage in Python applications in details https://github.com/bloomberg/memray <https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fbloomberg%2Fmemray&data=05%7C01%7Cjmorgan15%40crimson.ua.edu%7Cd0dae389465a4f20c3da08daab9838e5%7C2a00728ef0d040b4a4e8ce433f3fbca7%7C0%7C0%7C638010967575578774%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=lEW1ymCzHMuvTHoFMplFKZ4sQZNuy%2FDJsBjyXZV7LwE%3D&reserved=0>
cheers Clément
On 11 Oct 2022, at 05:48, Anne Noer Kolborg via yt-users <yt-users@python.org> wrote:
Dear yt community,
I am have encountered an out-of-memory problem (potential memory leak) when using yt to load and process several Ramses outputs in a row. Each simulation output is approximately 3 GB, but the RAM usage easily exceeds 16 GB in ~30 mins of computation time (single threaded computation time). When allocated more RAM the computation continues for longer but ultimately always runs out the memory before the end of the data-processing.
I have created a minimal working example that triggers this problem. It is uploaded here:http://paste.yt-project.org/show/404/ <https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Fpaste.yt-project.org%2Fshow%2F404%2F&data=05%7C01%7Cjmorgan15%40crimson.ua.edu%7Cd0dae389465a4f20c3da08daab9838e5%7C2a00728ef0d040b4a4e8ce433f3fbca7%7C0%7C0%7C638010967575578774%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=LOQzTjGIcN5eEDqCOEswtvVmMjpK%2FopplosYkOiGQ%2Fo%3D&reserved=0>
Log of the memory usage over time while running that script is here:http://paste.yt-project.org/show/406/ <https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Fpaste.yt-project.org%2Fshow%2F405%2F&data=05%7C01%7Cjmorgan15%40crimson.ua.edu%7Cd0dae389465a4f20c3da08daab9838e5%7C2a00728ef0d040b4a4e8ce433f3fbca7%7C0%7C0%7C638010967575578774%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=if%2Fm4vJC5RHVAD4NmUCIxPD2nQ6Gan0LTb8%2FmvNS0oA%3D&reserved=0>
As you can see the total memory usage simply grows for each round of the loop. I suspect that the memory allocated to the dataset does not get cleared.
Simulation output to test the code is uploaded here:http://use.yt/upload/1ace6072 <https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Fuse.yt%2Fupload%2F1ace6072&data=05%7C01%7Cjmorgan15%40crimson.ua.edu%7Cd0dae389465a4f20c3da08daab9838e5%7C2a00728ef0d040b4a4e8ce433f3fbca7%7C0%7C0%7C638010967575891276%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=YVUUX4ANsy2WoMcnY5OLEUxJfxSg9lMSQM2sbtC%2Bxf8%3D&reserved=0>
The del and clear_data() statements in line 18 and 19 do not impact the memory usage when commented in.
I tested the same bit of code using the “HiresIsolatedGalaxy” Enzo dataset from the sample datasets (https://yt-project.org/data/ <https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fyt-project.org%2Fdata%2F&data=05%7C01%7Cjmorgan15%40crimson.ua.edu%7Cd0dae389465a4f20c3da08daab9838e5%7C2a00728ef0d040b4a4e8ce433f3fbca7%7C0%7C0%7C638010967575891276%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=24oomFsnD2dZj0ZCVHpqVbzZ9RZfTO458koQ05PqgFQ%3D&reserved=0>). It is similar in size to the outputs I am trying to process and runs without issues. Therefore, I suspect it might be the ramses frontend causing the memory leak.
Any advice or pointers on how to debug this further, alternative ways to handle this or suggestions for how to use yt more efficiently are most welcome!
Best regards, Anne
_ _ _Version lists: _ Yt version 4.1.dev, changeset = 39949fb6bfba. (cloned from GitHub about a week ago) - the stable version 4.0.1 also produces the same issue. Python version 3.8.12 Gcc 4.8.5 CentOS Linux release 7.9.2009 // Linux 3.10.0 (x86-64 architecture)
_______________________________________________ yt-users mailing list --yt-users@python.org To unsubscribe send an email toyt-users-leave@python.org https://mail.python.org/mailman3/lists/yt-users.python.org/ Member address:contact@cphyc.me
<OpenPGP_0xEEBF77E6D1B29577.asc>
participants (5)
-
Anne Noer Kolborg -
Clément Robert -
Corentin CADIOU -
Jacob Morgan -
turquoisea.tsai＠gmail.com