All,

On Wed, Oct 5, 2016 at 11:46 AM, Francesc Alted <faltet@gmail.com> wrote:
2016-10-05 8:45 GMT+02:00 srean <srean.list@gmail.com>:
Good discussion, but was surprised by the absence of numexpr in the discussion., given how relevant it (numexpr) is to the topic.

Is the goal to fold in the numexpr functionality (and beyond) into Numpy ?

Yes, the question about merging numexpr into numpy has been something that periodically shows up in this list.  I think mostly everyone agree that it is a good idea, but things are not so easy, and so far nobody provided a good patch for this.  Also, the fact that numexpr relies on grouping an expression by using a string (e.g. (y = ne.evaluate("x**3 + tanh(x**2) + 4")) does not play well with the way in that numpy evaluates expressions, so something should be suggested to cope with this too.

As Francesc said, Numexpr is going to get most of its power through grouping a series of operations so it can send blocks to the CPU cache and run the entire series of operations on the cache before returning the block to system memory.  If it was just used to back-end NumPy, it would only gain from the multi-threading portion inside each function call. I'm not sure how one would go about grouping successive numpy expressions without modifying the Python interpreter?  

I put a bit of effort into extending numexpr to use 4-byte word opcodes instead of 1-byte.  Progress has been very slow, however, due to time constraints, but I have most of the numpy data types (u[1-4], i[1-4], f[4,8], c[8,16], S[1-4], U[1-4]).  On Tuesday I finished writing a Python generator script that writes all the C-side opcode macros for opcodes.hpp.  Now I have about 900 opcodes, and this could easily grow into thousands if more functions are added, so I also built a reverse lookup tree (based on collections.defaultdict) for the Python-side of numexpr.

Robert

--
Robert McLeod, Ph.D.
Center for Cellular Imaging and Nano Analytics (C-CINA)
Biozentrum der Universität Basel
Mattenstrasse 26, 4058 Basel
Work: +41.061.387.3225
robert.mcleod@unibas.ch