Re: [Numpy-discussion] NumPy-Discussion Digest, Vol 84, Issue 19
Hi again Julian,
its not np.any that is slow in this case its np.array([A, B, C])
Not really. The 4 unique alternative implementations provided all work quickly and correctly as-is, without a need to jump through a strange idiomatic hoop, so the problem does indeed lie with the implementation approach used in the current numpy.any(). It is unreasonable to expect users to know and always prepare their data using the idiom you propose if they might want to use the .any() function. If quirky idioms are needed to fix the performance of .any(), they should be placed in the implementation of .any().
this has the same performance as your code: a = empty([3] list(A.shape)
How do you know? This isn't valid Python, it won't run. Please can you send me a working version I can benchmark and compare? Graeme. On Sep 5, 2013, at 8:52 PM, <numpy-discussion-request@scipy.org> wrote:
hi, its not np.any that is slow in this case its np.array([A, B, C])
np.dstack([A, B, C]) is better but writing it like this has the same performance as your code: a = empty([3] list(A.shape) a[0] = A>5; a[1] = B<2; a[2] = A>10; np.any(a, 0)
I'll check if creating an array from a sequence can be improved for this case.
From my previous mail:
this has the same performance as your code: a = empty([3] list(A.shape)
For anyone that is interested. I ran a benchmark on the code after Julian kindly provided me with a correction to the listing he posted.
a = empty([3] + list(A.shape)) a[0] = A>5; a[1] = B<2; a[2] = A>10; np.any(a, 0)
Julian also suggested trying the idiom "np.vstack([A,B,C])" instead of [A,B,C]. Revised benchmarks here. I've moved the [A>5, B<2, A>10] creation outside the timing loop in all cases since it was distorting results due to array creation, which shouldn't be part of the any() timing measurement. I'm also now using separate test arrays to avoid the possibility of side effects between tests of different functions. The following results are produced consistently: np.any() -> 2.68 s np.any() with Julian's first idiom above: -> 0.24s faa.any() (original version) -> 0.2s np.any() with vstack(): 0.14s faa.any() with vstack: 0.1s faa.any() without vstack: 0.08s (alternative faa implementations: 0.11-0.12s) Conclusion: fast_any_all is 30x faster than numpy.any() 1.7 fast_any_all is 43% faster than numpy.any() 1.7 with the vstack() idiom, which I understand to be the basis for the new approach in numpy.any() 1.8 development branch. I'd be really interested to see the benchmarks under the current 1.8 master branch of numpy. Please can someone try this and send me the file? # git clone https://github.com/gbb/numpy-fast-any-all.git (read the source code to make sure I'm not evil) # cd numpy-fast-any-all # python test_fast_any_all.py > BENCHMARK.txt Incidentally, this is an appropriate example of a case where a 'performance idiom' becomes a 'penalty idiom' unexpectedly when the underlying implementation changes (vstack). Thanks for your suggestions, Julian. Graeme.
participants (1)
-
Graeme B. Bell