Program uses twice as much memory in Python 3.6 than in Python 3.5

INADA Naoki songofacandy at gmail.com
Tue Mar 28 06:11:24 EDT 2017


I can't reproduce it.

I managed to install pyopencl and run the script.  It takes more than
2 hours, and uses only 7GB RAM.
Maybe, some faster backend for OpenCL is required?

I used Microsoft Azure Compute, Standard_A4m_v2 (4 cores, 32 GB
memory) instance.

More easy way to reproduce is needed...

> My best idea about what's going on at the moment is that memory
> fragmentation is worse in Python 3.6 for some reason. The virtual memory
> size indicates that a large address space is acquired, but the resident
> memory size is smaller indicating that not all of that address space is
> actually used. In fact, the code might be especially bad to fragmentation
> because it takes a lot of small NumPy arrays and concatenates them into
> larger arrays. But I'm still surprised that this is only a problem with
> Python 3.6 (if this hypothesis is correct).
>
> Jan

Generally speaking, VMM vs RSS doesn't mean fragmentation.
If RSS : total allocated memory ratio is bigger than 1.5, it may be
fragmentation.
And large VMM won't cause swap.  Only RSS is meaningful.


More information about the Python-list mailing list