Program uses twice as much memory in Python 3.6 than in Python 3.5
INADA Naoki
songofacandy at gmail.com
Tue Mar 28 06:11:24 EDT 2017
I can't reproduce it.
I managed to install pyopencl and run the script. It takes more than
2 hours, and uses only 7GB RAM.
Maybe, some faster backend for OpenCL is required?
I used Microsoft Azure Compute, Standard_A4m_v2 (4 cores, 32 GB
memory) instance.
More easy way to reproduce is needed...
> My best idea about what's going on at the moment is that memory
> fragmentation is worse in Python 3.6 for some reason. The virtual memory
> size indicates that a large address space is acquired, but the resident
> memory size is smaller indicating that not all of that address space is
> actually used. In fact, the code might be especially bad to fragmentation
> because it takes a lot of small NumPy arrays and concatenates them into
> larger arrays. But I'm still surprised that this is only a problem with
> Python 3.6 (if this hypothesis is correct).
>
> Jan
Generally speaking, VMM vs RSS doesn't mean fragmentation.
If RSS : total allocated memory ratio is bigger than 1.5, it may be
fragmentation.
And large VMM won't cause swap. Only RSS is meaningful.
More information about the Python-list
mailing list