[Numpy-discussion] lazy evaluation
njs at pobox.com
Tue Jun 5 09:58:18 EDT 2012
On Tue, Jun 5, 2012 at 12:55 PM, mark florisson
<markflorisson88 at gmail.com> wrote:
> It would be great if we implement the NEP listed above, but with a few
> extensions. I think Numpy should handle the lazy evaluation part, and
> determine when expressions should be evaluated, etc. However, for each
> user operation, Numpy will call back a user-installed hook
> implementing some interface, to allow various packages to provide
> their own hooks to evaluate vector operations however they want. This
> will include packages such as Theano, which could run things on the
> GPU, Numexpr, and in the future
> https://github.com/markflorisson88/minivect (which will likely have an
> LLVM backend in the future, and possibly integrated with Numba to
> allow inlining of numba ufuncs). The project above tries to bring
> together all the different array expression compilers together in a
> single framework, to provide efficient array expressions specialized
> for any data layout (nditer on steroids if you will, with SIMD,
> threaded and inlining capabilities).
A global hook sounds ugly and hard to control -- it's hard to tell
which operations should be deferred and which should be forced, etc.
While it would be less magical, I think a more explicit API would in
the end be easier to use... something like
a, b, c, d = deferred([a, b, c, d])
e = a + b * c # 'e' is a deferred object too
f = np.dot(e, d) # so is 'f'
g = force(f) # 'g' is an ndarray
But at that point, this could easily be an external library, right?
All we'd need from numpy would be some way for external types to
override the evaluation of ufuncs, np.dot, etc.? We've recently seen
several reasons to want that functionality, and it seems like
developing these "improved numexpr" ideas would be much easier if they
didn't require doing deep surgery to numpy itself...
More information about the NumPy-Discussion