[SciPy-User] Large Memory usage while doing median filter
Moore, Eric (NIH/NIDDK) [F]
eric.moore2 at nih.gov
Mon May 11 10:34:03 EDT 2015
From: Warren Weckesser [mailto:warren.weckesser at gmail.com]
Sent: Monday, May 11, 2015 9:54 AM
To: SciPy Users List
Subject: Re: [SciPy-User] Large Memory usage while doing median filter
On Mon, May 11, 2015 at 1:25 AM, Jerome Kieffer <Jerome.Kieffer at esrf.fr> wrote:
On Mon, 11 May 2015 00:56:29 +0530
Joe P Ninan <indiajoe at gmail.com> wrote:
> Hi,
> I was trying median_filter in scipy.ndimage.filters
> on a 1024x1024 array.
>
> What I noticed is that the memory requirement grows really fast when we
> increase the size of the median filter.
> On a machine with 6gb RAM I could do only (150,150) size filter.
> Anything above gives Memory Error.
>
> On a bigger server I could see it takes about 16gb RAM while using a filter
> size (200, 200)
>
> I can understand, computation time increasing with size of filter, but why
> is the memory size exploding with respect to size of the median filter?
> Is this expected behaviour?
I guess this is because scipy creates a 1024x1024x(40000) array to do the sort along the last axis.
maybe no the best from the memorry point of view.
Maybe I didn't search hard enough, but I don't see where such an array is allocated. There are several layers of calls, from python in ndimage/filters.py down to C in ndimage/src/ni_filters.c, so maybe I missed it. Can you point to where such an array is created, or was that really a guess?
Warren
The really large array is allocated in NI_InitFilterOffsets, on line 518, in ni_support.c which is called from line 726 of ni_filter.c, in Ni_RankFilter.
For me, calling ndimage.median_filter(arr, 150), with arr a (1024, 1024) array of doubles or floats results in an allocation of 4050000000 bytes ( 3.77 GB). Which seems a little bit bigger than we would like here.
-Eric
More information about the SciPy-User
mailing list