[Numpy-discussion] numexpr with the new iterator

[Francesc Alted]

A Sunday 09 January 2011 23:45:02 Mark Wiebe escrigué:
> As a benchmark of C-based iterator usage and to make it work properly
> in a multi-threaded context, I've updated numexpr to use the new
> iterator.  In addition to some performance improvements, this also
> made it easy to add optional out= and order= parameters to the
> evaluate function.  The numexpr repository with this update is
> available here:
> 
> https://github.com/m-paradox/numexpr
> 
> To use it, you need the new_iterator branch of NumPy from here:
> 
> https://github.com/m-paradox/numpy
> 
> In all cases tested, the iterator version of numexpr's evaluate
> function matches or beats the standard version.  The timing results
> are below, with some explanatory comments placed inline:
[clip]

Your patch looks mostly fine to my eyes; good job!  Unfortunately, I've 
been unable to compile your new_iterator branch of NumPy:

numpy/core/src/multiarray/multiarraymodule.c:45:33: fatal error: 
new_iterator_pywrap.h: El fitxer o directori no existeix

Apparently, you forgot to add the new_iterator_pywrap.h file.

My idea would be to merge your patch in numexpr and make the new 
`evaluate_iter()` the default (i.e. make it `evaluate()`).  However, by 
looking into the code, it seems to me that unaligned arrays (this is an 
important use case when operating with columns of structured arrays) may 
need more fine-tuning for Intel platforms.  When I can compile the 
new_iterator branch, I'll give a try at unaligned data benchs.  

Also, I'd like to try out the new thread scheduling that you suggested 
to me privately (i.e. T0T1T0T1...  vs T0T0...T1T1...).

Thanks!

-- 
Francesc Alted