[Numpy-discussion] np.memmap and memory usage

[David Cournapeau]

On Wed, Jul 1, 2009 at 6:14 PM, Pauli Virtanen<pav at iki.fi> wrote:
> Wed, 01 Jul 2009 10:17:51 +0200, Emmanuelle Gouillart kirjoitti:
>>       I'm using numpy.memmap to open big 3-D arrays of Xray tomography
>> data. After I have created a new array using memmap, I modify the
>> contrast of every Z-slice (along the first dimension) inside a for loop,
>> for a better visualization of the data. Although I call memmap.flush
>> after each modification of a Z-slice, the memory used by Ipython keeps
>> increasing at every new iteration. At the end of the loop, the memory
>> used by Ipython is of the order of magnitude of the size of the data
>> file (1.8Go !). I would have expected that the maximum amount of memory
>> used would corresponde to only one Z-slice of the 3D array. See the code
>> snapshots below for more details.
>>
>>       Is this an expected behaviour? How can I reduce the amount of
>> memory used by Ipython and still process my data?
>
> How do you measure the memory used? Note that on Linux, "top" includes
> the size of OS caches for the memmap in the RSS size of a process.
> You can try to monitor "free" instead:

A good tool on linux is exmap:

http://ktown.kde.org/~seli/memory/analysis.html

Unfortunately, it is static, but more accurate than free or top.

David