[pypy-issue] Issue #2974: GC + an api to Malloc's call to ReleaseFreeMemory for the “unreturned memory” case (pypy/pypy)

Alex Kashirin issues-reply at bitbucket.org
Sat Mar 16 00:06:32 EDT 2019


New issue 2974: GC + an api to Malloc's call to ReleaseFreeMemory for the “unreturned memory” case
https://bitbucket.org/pypy/pypy/issues/2974/gc-an-api-to-mallocs-call-to

Alex Kashirin:

In this case, the none GLIBC_MALLOC is TCMALLOC 

A successful build with TCMALLOC goes well with this flag preset to the translation and build,


```
#!bash

LDFLAGS="-DTCMALLOC -ltcmalloc -lunwind \
-fno-builtin-malloc -fno-builtin-calloc -fno-builtin-realloc -fno-builtin-free" \
CFLAGS="$LDFLAGS" \
python ../../rpython/bin/rpython \
--no-shared --thread --make-jobs=8 \
--opt=jit targetpypystandalone.py --allworkingmodules 

```
for ref to full build, https://github.com/kashirin-alex/environments-builder/blob/master/scripts/pypy3.sh#L11

By the attached test,
The result are great, with at peak of 24GB~, the final RSS is 128MB for a steady consumption of 64MB while initial RSS was 56 MB

For the test purpose, the test command was :

```
#!bash

TCMALLOC_RELEASE_RATE=1000 MALLOCSTATS=2 pypy3 mem_chk.py

```
TCMALLOC_RELEASE_RATE, 1000 is a quite high, evaluated to release 100% each time possible. while 100 is around 10% of possible.


What could be a real improvement is an API in the PYPY-GC to Malloc's release mem calls,
In case of TCMALLOC , it is the 

for a compilation with the compiler definers, -DTCMALLOC or -DTCMALLOC_MINIMAL
required to add to the PYPY-GC with a corresponding py-api

```
#!C

#if defined(TCMALLOC) || defined(TCMALLOC_MINIMAL)
#include <gperftools/malloc_extension.h>
void ...pypy..GC.. malloc_release(){
   MallocExtension::instance()->ReleaseFreeMemory();
}
#endif
```

This way the memory can be release as needed and not only at RELEASE_RATE
  

the memtest.py results look as follow:

```
#!bash
Total memory consumed:
    GC used:            15.1MB (peak: 24473.6MB)
       in arenas:            4.7MB
       rawmalloced:          2.4MB
       nursery:              8.0MB
    raw assembler used: 57.1kB
    -----------------------------
    Total:              15.2MB

    Total memory allocated:
    GC allocated:            16.4MB (peak: 24474.1MB)
       in arenas:            5.6MB
       rawmalloced:          24460.1MB
       nursery:              8.0MB
    raw assembler allocated: 1.0MB
    -----------------------------
    Total:                   17.4MB

    Total time spent in GC:  11.225

********************************************************************************
steady consumed duration: 22.838454008102417
steady num tests: 207
test load duration: 54.853511571884155
initial_rss: 0.056 GB
actual consumed: 0.119 GB
********************************************************************************
------------------------------------------------
MALLOC:       19724440 (   18.8 MiB) Bytes in use by application
MALLOC: +     88956928 (   84.8 MiB) Bytes in page heap freelist
MALLOC: +       479624 (    0.5 MiB) Bytes in central cache freelist
MALLOC: +            0 (    0.0 MiB) Bytes in transfer cache freelist
MALLOC: +       849376 (    0.8 MiB) Bytes in thread cache freelists
MALLOC: +     39059456 (   37.2 MiB) Bytes in malloc metadata
MALLOC:   ------------
MALLOC: =    149069824 (  142.2 MiB) Actual memory used (physical + swap)
MALLOC: +  28540633088 (27218.5 MiB) Bytes released to OS (aka unmapped)
MALLOC:   ------------
MALLOC: =  28689702912 (27360.6 MiB) Virtual address space used
MALLOC:
MALLOC:            347              Spans in use
MALLOC:              2              Thread heaps in use
MALLOC:           8192              Tcmalloc page size
------------------------------------------------
Call ReleaseFreeMemory() to release freelist memory to the OS (via madvise()).
Bytes released to the OS take up virtual address space but no physical memory.
```

If you would like to have the pypy3 compilation in a package , let me know, it require some  .




More information about the pypy-issue mailing list