On Wed, Nov 21, 2018 at 3:19 PM Aronne Merrelli <aronne.merrelli@gmail.com> wrote:
Hello,
I am having a problem with the LinearNDInterpolator in SciPy, where it looks like there is a memory leak. I have a simple example here, all I do is setup a very small LinearNDInterpolator object and call the interpolator many times in a loop:
https://gist.github.com/aronnem/707dfeec07b74888fb7390d828684184
Note that if the "pts_list" and "vals" are swapped with the integer versions, there is no leak. Using the floats seems to produce the leak. I have tried both conda and Canopy installed environments, they all seem to have the same behavior. For example, here is the output I get with a scipy 1.1.0 compiled with Cython 0.28.5, numpy 1.13.3 on Mac OS X:
I can reproduce that as well with numpy 1.14.3, cython 0.29.0, scipy master.
rss_used= 4 KB vms_used= 0 KB rss_used= 2516 KB vms_used= 2560 KB rss_used= 4836 KB vms_used= 5188 KB rss_used= 6880 KB vms_used= 6664 KB rss_used= 9408 KB vms_used= 10248 KB rss_used= 11680 KB vms_used= 12076 KB rss_used= 14356 KB vms_used= 15872 KB rss_used= 16736 KB vms_used= 17792 KB rss_used= 19416 KB vms_used= 20496 KB rss_used= 22100 KB vms_used= 23472 KB
The only workaround I have found so far is to downgrade my numpy version to 1.11.3. So that seems to suggest it is a numpy problem; however this function is implemented in cython, so I am not sure how to tell whether this is a numpy, scipy, or cython problem.
It does look like a real memory leak, not the memory allocator holding on to allocations for longer than it has to. Given that numpy 1.11.3 didn't show this behavior, it would be helpful to use git bisect on the range numy v1.11.3-v1.13.3 to find when the numpy behavior changed. I just checked that v1.11.3 still builds with Cython 0.29.0, so I would suggest cython 0.29.0 and scipy master. Depending on your OS, you may need this small patch for older numpy: diff --git a/numpy/core/src/multiarray/numpyos.c b/numpy/core/src/multiarray/numpyos.c index 450ec40b6..ec1e5bc2c 100644 --- a/numpy/core/src/multiarray/numpyos.c +++ b/numpy/core/src/multiarray/numpyos.c @@ -15,7 +15,9 @@ #ifdef HAVE_STRTOLD_L #include <stdlib.h> -#include <xlocale.h> +#ifdef HAVE_XLOCALE_H + #include <xlocale.h> +#endif #endif Cheers, Ralf