[Numpy-discussion] numexpr with the new iterator

Francesc Alted faltet at pytables.org
Mon Jan 10 05:05:27 EST 2011

A Sunday 09 January 2011 23:45:02 Mark Wiebe escrigué:
> As a benchmark of C-based iterator usage and to make it work properly
> in a multi-threaded context, I've updated numexpr to use the new
> iterator.  In addition to some performance improvements, this also
> made it easy to add optional out= and order= parameters to the
> evaluate function.  The numexpr repository with this update is
> available here:
> https://github.com/m-paradox/numexpr
> To use it, you need the new_iterator branch of NumPy from here:
> https://github.com/m-paradox/numpy
> In all cases tested, the iterator version of numexpr's evaluate
> function matches or beats the standard version.  The timing results
> are below, with some explanatory comments placed inline:

Your patch looks mostly fine to my eyes; good job!  Unfortunately, I've 
been unable to compile your new_iterator branch of NumPy:

numpy/core/src/multiarray/multiarraymodule.c:45:33: fatal error: 
new_iterator_pywrap.h: El fitxer o directori no existeix

Apparently, you forgot to add the new_iterator_pywrap.h file.

My idea would be to merge your patch in numexpr and make the new 
`evaluate_iter()` the default (i.e. make it `evaluate()`).  However, by 
looking into the code, it seems to me that unaligned arrays (this is an 
important use case when operating with columns of structured arrays) may 
need more fine-tuning for Intel platforms.  When I can compile the 
new_iterator branch, I'll give a try at unaligned data benchs.  

Also, I'd like to try out the new thread scheduling that you suggested 
to me privately (i.e. T0T1T0T1...  vs T0T0...T1T1...).


Francesc Alted

More information about the NumPy-Discussion mailing list