Is evaluate_iter basically numpexpr but using your numpy branch or are there other changes?<br><br><div class="gmail_quote">On Sun, Jan 9, 2011 at 2:45 PM, Mark Wiebe <span dir="ltr"><<a href="mailto:mwwiebe@gmail.com">mwwiebe@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">As a benchmark of C-based iterator usage and to make it work properly in a multi-threaded context, I've updated numexpr to use the new iterator. In addition to some performance improvements, this also made it easy to add optional out= and order= parameters to the evaluate function. The numexpr repository with this update is available here:<div>
<br></div><div><a href="https://github.com/m-paradox/numexpr" target="_blank">https://github.com/m-paradox/numexpr</a></div><div><br></div><div>To use it, you need the new_iterator branch of NumPy from here:</div>
<div><br></div><div><a href="https://github.com/m-paradox/numpy" target="_blank">https://github.com/m-paradox/numpy</a></div><div><br></div><div>In all cases tested, the iterator version of numexpr's evaluate function matches or beats the standard version. The timing results are below, with some explanatory comments placed inline:</div>
<div><br></div><div>-Mark</div><div><br></div><div><div>In [1]: import numexpr as ne</div><div><br></div><div># numexpr front page example</div><div><br></div><div>In [2]: a = np.arange(1e6)</div><div>In [3]: b = np.arange(1e6)</div>
<div><br></div><div>In [4]: timeit a**2 + b**2 + 2*a*b</div><div>1 loops, best of 3: 121 ms per loop</div><div><br></div><div>In [5]: ne.set_num_threads(1)</div><div><br></div><div># iterator version performance matches standard version</div>
<div><br></div><div>In [6]: timeit ne.evaluate("a**2 + b**2 + 2*a*b")</div><div>10 loops, best of 3: 24.8 ms per loop</div><div>In [7]: timeit ne.evaluate_iter("a**2 + b**2 + 2*a*b")</div><div>10 loops, best of 3: 24.3 ms per loop</div>
<div><br></div><div>In [8]: ne.set_num_threads(2)</div><div><br></div><div># iterator version performance matches standard version</div><div><br></div><div>In [9]: timeit ne.evaluate("a**2 + b**2 + 2*a*b")</div>
<div>10 loops, best of 3: 21 ms per loop</div><div>In [10]: timeit ne.evaluate_iter("a**2 + b**2 + 2*a*b")</div><div>10 loops, best of 3: 20.5 ms per loop</div><div><br></div><div># numexpr front page example with a 10x bigger array</div>
<div><br></div><div>In [11]: a = np.arange(1e7)</div><div>In [12]: b = np.arange(1e7)</div><div><br></div><div>In [13]: ne.set_num_threads(2)</div><div><br></div><div># the iterator version performance improvement is due to</div>
<div># a small task scheduler tweak</div><div><br></div><div>In [14]: timeit ne.evaluate("a**2 + b**2 + 2*a*b")</div><div>1 loops, best of 3: 282 ms per loop</div><div>In [15]: timeit ne.evaluate_iter("a**2 + b**2 + 2*a*b")</div>
<div>1 loops, best of 3: 255 ms per loop</div><div><br></div><div># numexpr front page example with a Fortran contiguous array</div><div><br></div><div>In [16]: a = np.arange(1e7).reshape(10,100,100,100).T</div><div>In [17]: b = np.arange(1e7).reshape(10,100,100,100).T</div>
<div><br></div><div>In [18]: timeit a**2 + b**2 + 2*a*b</div><div>1 loops, best of 3: 3.22 s per loop</div><div><br></div><div>In [19]: ne.set_num_threads(1)</div><div><br></div><div># even with a C-ordered output, the iterator version performs better</div>
<div><br></div><div>In [20]: timeit ne.evaluate("a**2 + b**2 + 2*a*b")</div><div>1 loops, best of 3: 3.74 s per loop</div><div>In [21]: timeit ne.evaluate_iter("a**2 + b**2 + 2*a*b")</div><div>1 loops, best of 3: 379 ms per loop</div>
<div>In [22]: timeit ne.evaluate_iter("a**2 + b**2 + 2*a*b", order='C')</div><div>1 loops, best of 3: 2.03 s per loop</div><div><br></div><div>In [23]: ne.set_num_threads(2)</div><div><br></div><div># the standard version just uses 1 thread here, I believe</div>
<div># the iterator version performs the same as for the flat 1e7-sized array</div><div><br></div><div>In [24]: timeit ne.evaluate("a**2 + b**2 + 2*a*b")</div><div>1 loops, best of 3: 3.92 s per loop</div><div>
In [25]: timeit ne.evaluate_iter("a**2 + b**2 + 2*a*b")</div>
<div>1 loops, best of 3: 254 ms per loop</div><div>In [26]: timeit ne.evaluate_iter("a**2 + b**2 + 2*a*b", order='C')</div><div>1 loops, best of 3: 1.74 s per loop</div></div><div><br></div>
<br>_______________________________________________<br>
NumPy-Discussion mailing list<br>
<a href="mailto:NumPy-Discussion@scipy.org">NumPy-Discussion@scipy.org</a><br>
<a href="http://mail.scipy.org/mailman/listinfo/numpy-discussion" target="_blank">http://mail.scipy.org/mailman/listinfo/numpy-discussion</a><br>
<br></blockquote></div><br>