[Numpy-discussion] Strange memory consumption in numpy?

Martin Raspaud martin.raspaud at smhi.se
Thu May 16 03:35:22 EDT 2013


Hi all,

In the context of memory profiling an application (with memory_profiler
module) we came up a strange behaviour in numpy, see for yourselves:

Line #    Mem usage    Increment   Line Contents
================================================
    29                             @profile
    30    23.832 MB     0.000 MB   def main():
    31    46.730 MB    22.898 MB       arr1 = np.random.rand(1000000, 3)
    32    58.180 MB    11.449 MB       arr1s = arr1.astype(np.float32)
    33    35.289 MB   -22.891 MB       del arr1
    34    35.289 MB     0.000 MB       gc.collect()
    35    58.059 MB    22.770 MB       arr2 = np.random.rand(1000000, 3)
    36    69.500 MB    11.441 MB       arr2s = arr2.astype(np.float32)
    37    69.500 MB     0.000 MB       del arr2
    38    69.500 MB     0.000 MB       gc.collect()
    39    69.500 MB     0.000 MB       arr3 = np.random.rand(1000000, 3)
    40    80.945 MB    11.445 MB       arr3s = arr3.astype(np.float32)
    41    80.945 MB     0.000 MB       del arr3
    42    80.945 MB     0.000 MB       gc.collect()
    43    80.945 MB     0.000 MB       return arr1s, arr2s, arr3s


The lines 31-34 are behaving as expected, but then we don't understand
35-38 (why is arr2 not garbage collected ?) and 39-42 (why doesn't the
random allocate any memory ?).

Can anyone give a reasonable explanation ?

I attach the full script for reference.

Best regards,
Martin
-------------- next part --------------
A non-text attachment was scrubbed...
Name: testnumpymem.py
Type: text/x-python
Size: 1235 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130516/1f0d95dd/attachment.py>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: martin_raspaud.vcf
Type: text/x-vcard
Size: 303 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20130516/1f0d95dd/attachment.vcf>


More information about the NumPy-Discussion mailing list