memory leak in LinearNDInterpolator?
Hello, I am having a problem with the LinearNDInterpolator in SciPy, where it looks like there is a memory leak. I have a simple example here, all I do is setup a very small LinearNDInterpolator object and call the interpolator many times in a loop: https://gist.github.com/aronnem/707dfeec07b74888fb7390d828684184 Note that if the "pts_list" and "vals" are swapped with the integer versions, there is no leak. Using the floats seems to produce the leak. I have tried both conda and Canopy installed environments, they all seem to have the same behavior. For example, here is the output I get with a scipy 1.1.0 compiled with Cython 0.28.5, numpy 1.13.3 on Mac OS X: rss_used= 4 KB vms_used= 0 KB rss_used= 2516 KB vms_used= 2560 KB rss_used= 4836 KB vms_used= 5188 KB rss_used= 6880 KB vms_used= 6664 KB rss_used= 9408 KB vms_used= 10248 KB rss_used= 11680 KB vms_used= 12076 KB rss_used= 14356 KB vms_used= 15872 KB rss_used= 16736 KB vms_used= 17792 KB rss_used= 19416 KB vms_used= 20496 KB rss_used= 22100 KB vms_used= 23472 KB The only workaround I have found so far is to downgrade my numpy version to 1.11.3. So that seems to suggest it is a numpy problem; however this function is implemented in cython, so I am not sure how to tell whether this is a numpy, scipy, or cython problem. Regards, Aronne
On Thu, Nov 22, 2018 at 12:19 AM Aronne Merrelli <aronne.merrelli@gmail.com> wrote:
Hello,
I am having a problem with the LinearNDInterpolator in SciPy, where it looks like there is a memory leak. I have a simple example here, all I do is setup a very small LinearNDInterpolator object and call the interpolator many times in a loop:
https://gist.github.com/aronnem/707dfeec07b74888fb7390d828684184
Note that if the "pts_list" and "vals" are swapped with the integer versions, there is no leak. Using the floats seems to produce the leak. I have tried both conda and Canopy installed environments, they all seem to have the same behavior. For example, here is the output I get with a scipy 1.1.0 compiled with Cython 0.28.5, numpy 1.13.3 on Mac OS X:
I can reproduce what you see on scipy 1.1.0, cython 0.28.2, numpy 1.15.4 on debian. Also, the behaviour seems to depend on the type of `vals`; leaving `pts_list` as floats doesn't trigger the problem. András
On Wed, Nov 21, 2018 at 3:19 PM Aronne Merrelli <aronne.merrelli@gmail.com> wrote:
Hello,
I am having a problem with the LinearNDInterpolator in SciPy, where it looks like there is a memory leak. I have a simple example here, all I do is setup a very small LinearNDInterpolator object and call the interpolator many times in a loop:
https://gist.github.com/aronnem/707dfeec07b74888fb7390d828684184
Note that if the "pts_list" and "vals" are swapped with the integer versions, there is no leak. Using the floats seems to produce the leak. I have tried both conda and Canopy installed environments, they all seem to have the same behavior. For example, here is the output I get with a scipy 1.1.0 compiled with Cython 0.28.5, numpy 1.13.3 on Mac OS X:
I can reproduce that as well with numpy 1.14.3, cython 0.29.0, scipy master.
rss_used= 4 KB vms_used= 0 KB rss_used= 2516 KB vms_used= 2560 KB rss_used= 4836 KB vms_used= 5188 KB rss_used= 6880 KB vms_used= 6664 KB rss_used= 9408 KB vms_used= 10248 KB rss_used= 11680 KB vms_used= 12076 KB rss_used= 14356 KB vms_used= 15872 KB rss_used= 16736 KB vms_used= 17792 KB rss_used= 19416 KB vms_used= 20496 KB rss_used= 22100 KB vms_used= 23472 KB
The only workaround I have found so far is to downgrade my numpy version to 1.11.3. So that seems to suggest it is a numpy problem; however this function is implemented in cython, so I am not sure how to tell whether this is a numpy, scipy, or cython problem.
It does look like a real memory leak, not the memory allocator holding on to allocations for longer than it has to. Given that numpy 1.11.3 didn't show this behavior, it would be helpful to use git bisect on the range numy v1.11.3-v1.13.3 to find when the numpy behavior changed. I just checked that v1.11.3 still builds with Cython 0.29.0, so I would suggest cython 0.29.0 and scipy master. Depending on your OS, you may need this small patch for older numpy: diff --git a/numpy/core/src/multiarray/numpyos.c b/numpy/core/src/multiarray/numpyos.c index 450ec40b6..ec1e5bc2c 100644 --- a/numpy/core/src/multiarray/numpyos.c +++ b/numpy/core/src/multiarray/numpyos.c @@ -15,7 +15,9 @@ #ifdef HAVE_STRTOLD_L #include <stdlib.h> -#include <xlocale.h> +#ifdef HAVE_XLOCALE_H + #include <xlocale.h> +#endif #endif Cheers, Ralf
On Sat, Nov 24, 2018 at 1:20 PM Ralf Gommers <ralf.gommers@gmail.com> wrote:
On Wed, Nov 21, 2018 at 3:19 PM Aronne Merrelli <aronne.merrelli@gmail.com> wrote:
Hello,
I am having a problem with the LinearNDInterpolator in SciPy, where it looks like there is a memory leak. I have a simple example here, all I do is setup a very small LinearNDInterpolator object and call the interpolator many times in a loop:
https://gist.github.com/aronnem/707dfeec07b74888fb7390d828684184
Note that if the "pts_list" and "vals" are swapped with the integer versions, there is no leak. Using the floats seems to produce the leak. I have tried both conda and Canopy installed environments, they all seem to have the same behavior. For example, here is the output I get with a scipy 1.1.0 compiled with Cython 0.28.5, numpy 1.13.3 on Mac OS X:
I can reproduce that as well with numpy 1.14.3, cython 0.29.0, scipy master.
rss_used= 4 KB vms_used= 0 KB rss_used= 2516 KB vms_used= 2560 KB rss_used= 4836 KB vms_used= 5188 KB rss_used= 6880 KB vms_used= 6664 KB rss_used= 9408 KB vms_used= 10248 KB rss_used= 11680 KB vms_used= 12076 KB rss_used= 14356 KB vms_used= 15872 KB rss_used= 16736 KB vms_used= 17792 KB rss_used= 19416 KB vms_used= 20496 KB rss_used= 22100 KB vms_used= 23472 KB
The only workaround I have found so far is to downgrade my numpy version to 1.11.3. So that seems to suggest it is a numpy problem; however this function is implemented in cython, so I am not sure how to tell whether this is a numpy, scipy, or cython problem.
It does look like a real memory leak, not the memory allocator holding on to allocations for longer than it has to. Given that numpy 1.11.3 didn't show this behavior, it would be helpful to use git bisect on the range numy v1.11.3-v1.13.3 to find when the numpy behavior changed. I just checked that v1.11.3 still builds with Cython 0.29.0, so I would suggest cython 0.29.0 and scipy master.
Hi - Thank you both for confirming. I did manage to use bisect. All versions compiled without needing Ralf's small patch. The actual bisected commits aren't really relevant, because it looks like the behavior is controlled by value of the NPY_RELAXED_STRIDES_CHECKING variable. This environment variable changed to default to 1/True in numpy 1.12.1, if it was not set in the shell. In earlier versions, it defaulted to 0/False if not set in the shell. It also looks like I can prevent the memory leak behavior in newer numpy (at least, I checked 1.12.1 and 1.15.4), if I rebuild numpy NPY_RELAXED_STRIDES_CHECKING=0. I do not understand what that variable is controlling, so I am still not sure if this is numpy, scipy, or cython issue. Anyone have a suggestion? Otherwise, at least I've found a relatively simple workaround! thanks Aronne
Depending on your OS, you may need this small patch for older numpy:
diff --git a/numpy/core/src/multiarray/numpyos.c b/numpy/core/src/multiarray/numpyos.c index 450ec40b6..ec1e5bc2c 100644 --- a/numpy/core/src/multiarray/numpyos.c +++ b/numpy/core/src/multiarray/numpyos.c @@ -15,7 +15,9 @@
#ifdef HAVE_STRTOLD_L #include <stdlib.h> -#include <xlocale.h> +#ifdef HAVE_XLOCALE_H + #include <xlocale.h> +#endif #endif
Cheers, Ralf
_______________________________________________ SciPy-Dev mailing list SciPy-Dev@python.org https://mail.python.org/mailman/listinfo/scipy-dev
On Sun, Nov 25, 2018 at 12:10 PM Aronne Merrelli <aronne.merrelli@gmail.com> wrote:
On Sat, Nov 24, 2018 at 1:20 PM Ralf Gommers <ralf.gommers@gmail.com> wrote:
On Wed, Nov 21, 2018 at 3:19 PM Aronne Merrelli < aronne.merrelli@gmail.com> wrote:
Hello,
I am having a problem with the LinearNDInterpolator in SciPy, where it looks like there is a memory leak. I have a simple example here, all I do is setup a very small LinearNDInterpolator object and call the interpolator many times in a loop:
https://gist.github.com/aronnem/707dfeec07b74888fb7390d828684184
Note that if the "pts_list" and "vals" are swapped with the integer versions, there is no leak. Using the floats seems to produce the leak. I have tried both conda and Canopy installed environments, they all seem to have the same behavior. For example, here is the output I get with a scipy 1.1.0 compiled with Cython 0.28.5, numpy 1.13.3 on Mac OS X:
I can reproduce that as well with numpy 1.14.3, cython 0.29.0, scipy master.
rss_used= 4 KB vms_used= 0 KB rss_used= 2516 KB vms_used= 2560 KB rss_used= 4836 KB vms_used= 5188 KB rss_used= 6880 KB vms_used= 6664 KB rss_used= 9408 KB vms_used= 10248 KB rss_used= 11680 KB vms_used= 12076 KB rss_used= 14356 KB vms_used= 15872 KB rss_used= 16736 KB vms_used= 17792 KB rss_used= 19416 KB vms_used= 20496 KB rss_used= 22100 KB vms_used= 23472 KB
The only workaround I have found so far is to downgrade my numpy version to 1.11.3. So that seems to suggest it is a numpy problem; however this function is implemented in cython, so I am not sure how to tell whether this is a numpy, scipy, or cython problem.
It does look like a real memory leak, not the memory allocator holding on to allocations for longer than it has to. Given that numpy 1.11.3 didn't show this behavior, it would be helpful to use git bisect on the range numy v1.11.3-v1.13.3 to find when the numpy behavior changed. I just checked that v1.11.3 still builds with Cython 0.29.0, so I would suggest cython 0.29.0 and scipy master.
Hi - Thank you both for confirming.
I did manage to use bisect. All versions compiled without needing Ralf's small patch. The actual bisected commits aren't really relevant, because it looks like the behavior is controlled by value of the NPY_RELAXED_STRIDES_CHECKING variable. This environment variable changed to default to 1/True in numpy 1.12.1, if it was not set in the shell. In earlier versions, it defaulted to 0/False if not set in the shell.
It also looks like I can prevent the memory leak behavior in newer numpy (at least, I checked 1.12.1 and 1.15.4), if I rebuild numpy NPY_RELAXED_STRIDES_CHECKING=0.
I do not understand what that variable is controlling, so I am still not sure if this is numpy, scipy, or cython issue. Anyone have a suggestion?
Otherwise, at least I've found a relatively simple workaround!
thanks Aronne
What happens if you set `NPY_RELAXED_STRIDES_DEBUG=1` with the problem compile? Chuck
On Sun, Nov 25, 2018 at 1:32 PM Charles R Harris <charlesr.harris@gmail.com> wrote:
On Sun, Nov 25, 2018 at 12:10 PM Aronne Merrelli < aronne.merrelli@gmail.com> wrote:
On Sat, Nov 24, 2018 at 1:20 PM Ralf Gommers <ralf.gommers@gmail.com> wrote:
On Wed, Nov 21, 2018 at 3:19 PM Aronne Merrelli < aronne.merrelli@gmail.com> wrote:
Hello,
I am having a problem with the LinearNDInterpolator in SciPy, where it looks like there is a memory leak. I have a simple example here, all I do is setup a very small LinearNDInterpolator object and call the interpolator many times in a loop:
https://gist.github.com/aronnem/707dfeec07b74888fb7390d828684184
Note that if the "pts_list" and "vals" are swapped with the integer versions, there is no leak. Using the floats seems to produce the leak. I have tried both conda and Canopy installed environments, they all seem to have the same behavior. For example, here is the output I get with a scipy 1.1.0 compiled with Cython 0.28.5, numpy 1.13.3 on Mac OS X:
I can reproduce that as well with numpy 1.14.3, cython 0.29.0, scipy master.
rss_used= 4 KB vms_used= 0 KB rss_used= 2516 KB vms_used= 2560 KB rss_used= 4836 KB vms_used= 5188 KB rss_used= 6880 KB vms_used= 6664 KB rss_used= 9408 KB vms_used= 10248 KB rss_used= 11680 KB vms_used= 12076 KB rss_used= 14356 KB vms_used= 15872 KB rss_used= 16736 KB vms_used= 17792 KB rss_used= 19416 KB vms_used= 20496 KB rss_used= 22100 KB vms_used= 23472 KB
The only workaround I have found so far is to downgrade my numpy version to 1.11.3. So that seems to suggest it is a numpy problem; however this function is implemented in cython, so I am not sure how to tell whether this is a numpy, scipy, or cython problem.
It does look like a real memory leak, not the memory allocator holding on to allocations for longer than it has to. Given that numpy 1.11.3 didn't show this behavior, it would be helpful to use git bisect on the range numy v1.11.3-v1.13.3 to find when the numpy behavior changed. I just checked that v1.11.3 still builds with Cython 0.29.0, so I would suggest cython 0.29.0 and scipy master.
Hi - Thank you both for confirming.
I did manage to use bisect. All versions compiled without needing Ralf's small patch. The actual bisected commits aren't really relevant, because it looks like the behavior is controlled by value of the NPY_RELAXED_STRIDES_CHECKING variable. This environment variable changed to default to 1/True in numpy 1.12.1, if it was not set in the shell. In earlier versions, it defaulted to 0/False if not set in the shell.
It also looks like I can prevent the memory leak behavior in newer numpy (at least, I checked 1.12.1 and 1.15.4), if I rebuild numpy NPY_RELAXED_STRIDES_CHECKING=0.
I do not understand what that variable is controlling, so I am still not sure if this is numpy, scipy, or cython issue. Anyone have a suggestion?
Otherwise, at least I've found a relatively simple workaround!
thanks Aronne
What happens if you set `NPY_RELAXED_STRIDES_DEBUG=1` with the problem compile?
Chuck
Hi - I do not see any change with the DEBUG flag enabled. This is with: Python 3.7.1, numpy 1.15.4, scipy 1.1.0, CentOS 6.1.
_______________________________________________ SciPy-Dev mailing list SciPy-Dev@python.org https://mail.python.org/mailman/listinfo/scipy-dev
participants (4)
-
Andras Deak -
Aronne Merrelli -
Charles R Harris -
Ralf Gommers