Dear devs, Sorry for posting a numpy-issue on the scipy-list. I am not (yet) subscribed to the numpy list, but I believe that the two community have a big enough overlap so it shouldn't matter too much where I post it. I recently encountered the issue that I found that np.any(x) is sort of veeeery slow for big arrays, even if every element has a non-zero value and I only need a True/False response. So my thinking was that if np.any(x) encounters the first non-zero value it should simply return True, which should take basically no time at all if every element in the array is non-zero. Searching for it I found the following two old issues on numpy: 1. https://github.com/numpy/numpy/issues/2269 It is an issue from 2010, but got some traffic again in 2016 and 2017. 2. https://github.com/numpy/numpy/issues/3446 This is a related issue from 2013, reporting a potential performance regression between numpy 1.6.2 and 1.7.0, that got some traffic in 2016 again as well. I just wanted to ask about the opinions of devs more familiar with these two functions (np.all(), np.any()). Would there be better ways to check if any element in a big (1D or higher dimensions)-array is non-zero (or the reverse with np.all)? Thanks, Dieter
Performing a test (to determine whether the loop should be aborted) after looking at each element would cause `np.any` to run slower if all elements are zero, or if the first non-zero element is near the end of the array. Possibly a separate method that operates in this fashion is needed? Phillip On Mon, Jul 2, 2018 at 11:03 AM, Dieter Werthmüller <dieter@werthmuller.org> wrote:
Dear devs,
Sorry for posting a numpy-issue on the scipy-list. I am not (yet) subscribed to the numpy list, but I believe that the two community have a big enough overlap so it shouldn't matter too much where I post it.
I recently encountered the issue that I found that np.any(x) is sort of veeeery slow for big arrays, even if every element has a non-zero value and I only need a True/False response. So my thinking was that if np.any(x) encounters the first non-zero value it should simply return True, which should take basically no time at all if every element in the array is non-zero.
Searching for it I found the following two old issues on numpy:
1. https://github.com/numpy/numpy/issues/2269
It is an issue from 2010, but got some traffic again in 2016 and 2017.
2. https://github.com/numpy/numpy/issues/3446
This is a related issue from 2013, reporting a potential performance regression between numpy 1.6.2 and 1.7.0, that got some traffic in 2016 again as well.
I just wanted to ask about the opinions of devs more familiar with these two functions (np.all(), np.any()). Would there be better ways to check if any element in a big (1D or higher dimensions)-array is non-zero (or the reverse with np.all)?
Thanks, Dieter
_______________________________________________ SciPy-Dev mailing list SciPy-Dev@python.org https://mail.python.org/mailman/listinfo/scipy-dev
There is an open issue for this since 2012: https://github.com/numpy/numpy/issues/2269 Some python and c solutions are suggested in that thread. Op wo 4 jul. 2018 om 06:44 schreef Phillip Feldman < phillip.m.feldman@gmail.com>:
Performing a test (to determine whether the loop should be aborted) after looking at each element would cause `np.any` to run slower if all elements are zero, or if the first non-zero element is near the end of the array. Possibly a separate method that operates in this fashion is needed?
Phillip
On Mon, Jul 2, 2018 at 11:03 AM, Dieter Werthmüller < dieter@werthmuller.org> wrote:
Dear devs,
Sorry for posting a numpy-issue on the scipy-list. I am not (yet) subscribed to the numpy list, but I believe that the two community have a big enough overlap so it shouldn't matter too much where I post it.
I recently encountered the issue that I found that np.any(x) is sort of veeeery slow for big arrays, even if every element has a non-zero value and I only need a True/False response. So my thinking was that if np.any(x) encounters the first non-zero value it should simply return True, which should take basically no time at all if every element in the array is non-zero.
Searching for it I found the following two old issues on numpy:
1. https://github.com/numpy/numpy/issues/2269
It is an issue from 2010, but got some traffic again in 2016 and 2017.
2. https://github.com/numpy/numpy/issues/3446
This is a related issue from 2013, reporting a potential performance regression between numpy 1.6.2 and 1.7.0, that got some traffic in 2016 again as well.
I just wanted to ask about the opinions of devs more familiar with these two functions (np.all(), np.any()). Would there be better ways to check if any element in a big (1D or higher dimensions)-array is non-zero (or the reverse with np.all)?
Thanks, Dieter
_______________________________________________ SciPy-Dev mailing list SciPy-Dev@python.org https://mail.python.org/mailman/listinfo/scipy-dev
_______________________________________________ SciPy-Dev mailing list SciPy-Dev@python.org https://mail.python.org/mailman/listinfo/scipy-dev
Sorry, I see you mention that issue. Anyway, I would suggest going for a cython or C workaround at the moment and commenting on the tickets. As Phillip says, a separate method would be needed as any and all do a reduce. Some methods like firstnotvalue and firstvalue, returning index of a difference from value or the value, so like https://pypi.org/project/py_find_1st/ mentioned in the ticket. For multidim arrays, as this operates on 1D arrays, perhaps the firstvalue applied per axis... Op wo 4 jul. 2018 om 09:43 schreef Benny Malengier < benny.malengier@gmail.com>:
There is an open issue for this since 2012: https://github.com/numpy/numpy/issues/2269
Some python and c solutions are suggested in that thread.
Op wo 4 jul. 2018 om 06:44 schreef Phillip Feldman < phillip.m.feldman@gmail.com>:
Performing a test (to determine whether the loop should be aborted) after looking at each element would cause `np.any` to run slower if all elements are zero, or if the first non-zero element is near the end of the array. Possibly a separate method that operates in this fashion is needed?
Phillip
On Mon, Jul 2, 2018 at 11:03 AM, Dieter Werthmüller < dieter@werthmuller.org> wrote:
Dear devs,
Sorry for posting a numpy-issue on the scipy-list. I am not (yet) subscribed to the numpy list, but I believe that the two community have a big enough overlap so it shouldn't matter too much where I post it.
I recently encountered the issue that I found that np.any(x) is sort of veeeery slow for big arrays, even if every element has a non-zero value and I only need a True/False response. So my thinking was that if np.any(x) encounters the first non-zero value it should simply return True, which should take basically no time at all if every element in the array is non-zero.
Searching for it I found the following two old issues on numpy:
1. https://github.com/numpy/numpy/issues/2269
It is an issue from 2010, but got some traffic again in 2016 and 2017.
2. https://github.com/numpy/numpy/issues/3446
This is a related issue from 2013, reporting a potential performance regression between numpy 1.6.2 and 1.7.0, that got some traffic in 2016 again as well.
I just wanted to ask about the opinions of devs more familiar with these two functions (np.all(), np.any()). Would there be better ways to check if any element in a big (1D or higher dimensions)-array is non-zero (or the reverse with np.all)?
Thanks, Dieter
_______________________________________________ SciPy-Dev mailing list SciPy-Dev@python.org https://mail.python.org/mailman/listinfo/scipy-dev
_______________________________________________ SciPy-Dev mailing list SciPy-Dev@python.org https://mail.python.org/mailman/listinfo/scipy-dev
Thanks Phillip and Benny for your replies. Yes, I specifically saw the issue you mentioned Benny. It is actually from 2010, and was moved to GitHub in 2012. This is why I thought I quickly ask on the mailing list if anything happened or is planned in this regard, because I imagine that very old issues are easily forgotten. It looks like this didn't change in the last 8 years, and probably won't in the (near) future. Which is fine, I just wanted to check before I search for other solutions. Thanks, Dieter On 04/07/18 02:56, Benny Malengier wrote:
Sorry, I see you mention that issue. Anyway, I would suggest going for a cython or C workaround at the moment and commenting on the tickets. As Phillip says, a separate method would be needed as any and all do a reduce. Some methods like firstnotvalue and firstvalue, returning index of a difference from value or the value, so like https://pypi.org/project/py_find_1st/ mentioned in the ticket.
For multidim arrays, as this operates on 1D arrays, perhaps the firstvalue applied per axis...
Op wo 4 jul. 2018 om 09:43 schreef Benny Malengier <benny.malengier@gmail.com <mailto:benny.malengier@gmail.com>>:
There is an open issue for this since 2012: https://github.com/numpy/numpy/issues/2269
Some python and c solutions are suggested in that thread.
Op wo 4 jul. 2018 om 06:44 schreef Phillip Feldman <phillip.m.feldman@gmail.com <mailto:phillip.m.feldman@gmail.com>>:
Performing a test (to determine whether the loop should be aborted) after looking at each element would cause `np.any` to run slower if all elements are zero, or if the first non-zero element is near the end of the array. Possibly a separate method that operates in this fashion is needed?
Phillip
On Mon, Jul 2, 2018 at 11:03 AM, Dieter Werthmüller <dieter@werthmuller.org <mailto:dieter@werthmuller.org>> wrote:
Dear devs,
Sorry for posting a numpy-issue on the scipy-list. I am not (yet) subscribed to the numpy list, but I believe that the two community have a big enough overlap so it shouldn't matter too much where I post it.
I recently encountered the issue that I found that np.any(x) is sort of veeeery slow for big arrays, even if every element has a non-zero value and I only need a True/False response. So my thinking was that if np.any(x) encounters the first non-zero value it should simply return True, which should take basically no time at all if every element in the array is non-zero.
Searching for it I found the following two old issues on numpy:
1. https://github.com/numpy/numpy/issues/2269
It is an issue from 2010, but got some traffic again in 2016 and 2017.
2. https://github.com/numpy/numpy/issues/3446
This is a related issue from 2013, reporting a potential performance regression between numpy 1.6.2 and 1.7.0, that got some traffic in 2016 again as well.
I just wanted to ask about the opinions of devs more familiar with these two functions (np.all(), np.any()). Would there be better ways to check if any element in a big (1D or higher dimensions)-array is non-zero (or the reverse with np.all)?
Thanks, Dieter
_______________________________________________ SciPy-Dev mailing list SciPy-Dev@python.org <mailto:SciPy-Dev@python.org> https://mail.python.org/mailman/listinfo/scipy-dev
_______________________________________________ SciPy-Dev mailing list SciPy-Dev@python.org <mailto:SciPy-Dev@python.org> https://mail.python.org/mailman/listinfo/scipy-dev
_______________________________________________ SciPy-Dev mailing list SciPy-Dev@python.org https://mail.python.org/mailman/listinfo/scipy-dev
participants (3)
-
Benny Malengier -
Dieter Werthmüller -
Phillip Feldman