[Tutor] Question about the memory manager

[Albert-Jan Roskam]

> From: eryksun at gmail.com
> Date: Thu, 14 Jan 2016 04:42:57 -0600
> Subject: Re: [Tutor] Question about the memory manager
> To: tutor at python.org
> CC: sjeik_appie at hotmail.com
> 
> On Thu, Jan 14, 2016 at 3:03 AM, Albert-Jan Roskam
> <sjeik_appie at hotmail.com> wrote:
> >
> > These two pages are quite nice. The author says the memory used by small objects is
> > never returned to the OS, which may be problematic for long running processes.
> 
> The article by Evan Jones discusses a patch to enable releasing unused
> arenas (i.e. "how the problem was fixed"). Starting with 2.5, unused
> arenas do get released back to the heap. Here's the diff in which Tim
> Peters merged in a "heavily altered derivative" of Evan's patch [1].
> 
> Also, 2.7 and 3.3 bypass C malloc/free and the process heap to instead
> use mmap/munmap on POSIX when available. This avoids the heap
> high-water mark problem. Similarly, 3.4 switched to using
> VirtualAlloc/VirtualFree on Windows. 3.4 also introduced the
> PyObjectArenaAllocator and associated C API functions [2] to allow
> modifying the default allocators.
> 
> [1]: https://hg.python.org/cpython/diff/685849bd905c/Objects/obmalloc.c
> [2]: https://docs.python.org/3/c-api/memory.html#customize-pyobject-arena-allocator

Hi Eryk,

Thanks a lot for the info and the links. This is truly interesting to read about! Glad to know that the high-water mark problem is no longer relevant anymore in recent Python versions. Also, thank you for your suggestion about psutils (and ctypes).

Best wishes,
Albert-Jan