I guess that INADA-san used pyperformance --track-memory.
pyperf --track-memory doc:
"--track-memory: get the memory peak usage. it is less accurate than
tracemalloc, but has a lower overhead. On Linux, compute the sum of
Private_Clean and Private_Dirty memory mappings of /proc/self/smaps.
On Windows, get PeakPagefileUsage of GetProcessMemoryInfo() (of the
current process): the peak value of the Commit Charge during the
lifetime of this process."
https://pyperf.readthedocs.io/en/latest/runner.html#misc
On Linux, pyperf uses read_smap_file() of pyperf._memory:
https://github.com/vstinner/pyperf/blob/master/pyperf/_memory.py
# Code to parse Linux /proc/%d/smaps files.
#
# See http://bmaurer.blogspot.com/2006/03/memory-usage-with-smaps.html for
# a quick introduction to smaps.
#
# Need Linux 2.6.16 or newer.
def read_smap_file():
total = 0
fp = open(proc_path("self/smaps"), "rb")
with fp:
for line in fp:
# Include both Private_Clean and Private_Dirty sections.
line = line.rstrip()
if line.startswith(b"Private_") and line.endswith(b'kB'):
parts = line.split()
total += int(parts[1]) * 1024
return total
It spawns a thread which reads /proc/self/smaps every milliseconds and
then report the *maximum*.
Victor
Le jeu. 4 juil. 2019 à 17:12, Tim Peters
[Antoine Pitrou
] Ah, interesting. Were you able to measure the memory footprint as well?
[Inada Naoki
] Hmm, it is not good. mimalloc uses MADV_FREE so it may affect to some benchmarks. I will look it later.
``` $ ./python -m pyperf compare_to pymalloc-mem.json mimalloc-mem.json -G Slower (60): - logging_format: 10.6 MB +- 384.2 kB -> 27.2 MB +- 21.3 kB: 2.58x slower (+158%) ...
Could you say more about what's being measured here? Like, if this is on Linux, is it reporting RSS? VSS? Something else?
mimalloc is "most like" obmalloc among those I've looked at in recent weeks. I noted before that its pools (they call them "pages") and arenas (called "segments") are at least 16x larger than obmalloc uses (64 KiB minimum pool/page size, and 4 MiB minimum arena/segment size, in mimalloc).
Like all "modern" 64-bit allocators, it cares little about reserving largish blobs of address space, so I'd _expect_ Linuxy VSS to zoom. But it also releases (in some sense, ;like MADV_FREE) physical RAM back to the system at a granularity far smaller than arena'segment.
At an extreme, the SuperMalloc I linked to earlier reserves a 512 MiB vector at the start, so no program linking to that can consume less than half a gig of address space. But it only expects to _use_ a few 4 KiB OS pages out of that. mimalloc doesn't do anything anywhere near _that_ gonzo (& since mimalloc comes out of Microsoft, it never will - "total virtual memory" on Windows is a system-wide resource, limited to the sum of physical RAM + pagefile size - no "overcommit" is allowed). _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/ES3BFCBX...
-- Night gathers, and now my watch begins. It shall not end until my death.