[Numpy-discussion] lazy evaluation

James Bergstra bergstrj at iro.umontreal.ca
Mon Jun 11 00:03:40 EDT 2012

Hi all, (sorry for missing the debate, I don't often check my
numpy-list folder.)

I agree that an "official" numpy solution to this problem is
premature, but at the same time I think the failure to approach
anything remotely resembling a consensus on how to deal with lazy
evaluation is really gumming up the works across the numpy community.
With apologies in advance to projects I don't cite (sorry!) it is
currently the case that many high-level libraries (e.g. pacal, pymc,
theano, sympy, pylearn2) offer more or less symbolic math features for
scientific applications, but each one defines it's own lazy-numpy AST
thing to track functional relationships between inputs and/or random
variables. At the same time, none of these ASTs (except arguably
Theano's) is handled natively by the many competing lazy-evaluation
compiler runtimes (e.g. cython, numba, theano, numexpr). So
consequently, the feature-specific ASTs often become more of a
performance *problem* than a part of the optimizing-compiler pathway
and libraries that provide end-user APIs (in my work I think of
sklearn and skimage) continue to "wait and see" and don't commit to
*any* of the options (except labour-intensive cython), so we all lose.

The interesting development/insight I got from numba's byte-code
parsing technique is the illustration that *Python byte code* is:

a) a standard data structure that all Python code is already using

b) editable (see e.g. http://code.google.com/p/byteplay)

c) in pretty direct correspondance with high level (e.g. Theano's)
"abstract" syntax graphs

d) an unambiguous and obvious program specification for optimization
(e.g. numba)

After a little proof of concept work, I think that many high-level
semantic features (e.g. turning a stochastic function into a sampler
via PyMC, tracking uncertainty through computations, or minimizing a
numpy function directly by automatic differentiation) can and should
be done as bytecode -> bytecode transforms.  An implementation of e.g.
auto-diff will have to recognize when it can (and cannot) make sense
of a code object... so functions with lots of control flow, yield
statements, exception handling and the like may just be rejected.
That's OK because mathematical code does often not require complex
(often even *any*) control flow constructs.

With regards to users being surprised by strange resource usage
levels... this surprise can be avoided because a user who is applying
such transforms will be well aware that he/she has transformed the
original function into a new function.  That transformation would be
explicit, so there will be little suggestion from the program syntax
that the new function  has any statements in common with the original.
The new function will have different statements, different resource
usage profile, etc. I think APIs for this sort of bytecode->bytecode
transformation can avoid surprising users if they are done right.

If anyone is interested in my ongoing API & bytecode adventure in why
/ how lazy computing could be useful, I've put together a few tiny
hypothetically-runnable examples here:


The purpose of the examples is to show how the features of e.g. Theano
and PyMC could be expressed as operators on raw Python code. Perhaps
most importantly of all, these transforms would work together: a PaCal
transform could automatically generate a likelihood function from a
model and data, and then a Theano transform could provide the
parameter gradients required to fit the likelihood. This natural
chaining is a complete PITA when every project uses its own AST.

That numba fork also includes very sketchy pseudocode of the main work
routines in the numba/ad.py and numba/rv.py files. The linear_svm
example was recently using Theano as a backend. I don't think it works
right now but FWIW it is still close to running.

Sorry for the long post,

- James

On Wed, Jun 6, 2012 at 5:22 AM, Dag Sverre Seljebotn
<d.s.seljebotn at astro.uio.no> wrote:
> On 06/06/2012 12:06 AM, mark florisson wrote:
>> On 5 June 2012 22:36, Dag Sverre Seljebotn<d.s.seljebotn at astro.uio.no>  wrote:
>>> On 06/05/2012 10:47 PM, mark florisson wrote:
>>>> On 5 June 2012 20:17, Nathaniel Smith<njs at pobox.com>    wrote:
>>>>> On Tue, Jun 5, 2012 at 7:08 PM, mark florisson
>>>>> <markflorisson88 at gmail.com>    wrote:
>>>>>> On 5 June 2012 17:38, Nathaniel Smith<njs at pobox.com>    wrote:
>>>>>>> On Tue, Jun 5, 2012 at 4:12 PM, mark florisson
>>>>>>> <markflorisson88 at gmail.com>    wrote:
>>>>>>>> On 5 June 2012 14:58, Nathaniel Smith<njs at pobox.com>    wrote:
>>>>>>>>> On Tue, Jun 5, 2012 at 12:55 PM, mark florisson
>>>>>>>>> <markflorisson88 at gmail.com>    wrote:
>>>>>>>>>> It would be great if we implement the NEP listed above, but with a few
>>>>>>>>>> extensions. I think Numpy should handle the lazy evaluation part, and
>>>>>>>>>> determine when expressions should be evaluated, etc. However, for each
>>>>>>>>>> user operation, Numpy will call back a user-installed hook
>>>>>>>>>> implementing some interface, to allow various packages to provide
>>>>>>>>>> their own hooks to evaluate vector operations however they want. This
>>>>>>>>>> will include packages such as Theano, which could run things on the
>>>>>>>>>> GPU, Numexpr, and in the future
>>>>>>>>>> https://github.com/markflorisson88/minivect (which will likely have an
>>>>>>>>>> LLVM backend in the future, and possibly integrated with Numba to
>>>>>>>>>> allow inlining of numba ufuncs). The project above tries to bring
>>>>>>>>>> together all the different array expression compilers together in a
>>>>>>>>>> single framework, to provide efficient array expressions specialized
>>>>>>>>>> for any data layout (nditer on steroids if you will, with SIMD,
>>>>>>>>>> threaded and inlining capabilities).
>>>>>>>>> A global hook sounds ugly and hard to control -- it's hard to tell
>>>>>>>>> which operations should be deferred and which should be forced, etc.
>>>>>>>> Yes, but for the user the difference should not be visible (unless
>>>>>>>> operations can raise exceptions, in which case you choose the safe
>>>>>>>> path, or let the user configure what to do).
>>>>>>>>> While it would be less magical, I think a more explicit API would in
>>>>>>>>> the end be easier to use... something like
>>>>>>>>>    a, b, c, d = deferred([a, b, c, d])
>>>>>>>>>    e = a + b * c  # 'e' is a deferred object too
>>>>>>>>>    f = np.dot(e, d)  # so is 'f'
>>>>>>>>>    g = force(f)  # 'g' is an ndarray
>>>>>>>>>    # or
>>>>>>>>>    force(f, out=g)
>>>>>>>>> But at that point, this could easily be an external library, right?
>>>>>>>>> All we'd need from numpy would be some way for external types to
>>>>>>>>> override the evaluation of ufuncs, np.dot, etc.? We've recently seen
>>>>>>>>> several reasons to want that functionality, and it seems like
>>>>>>>>> developing these "improved numexpr" ideas would be much easier if they
>>>>>>>>> didn't require doing deep surgery to numpy itself...
>>>>>>>> Definitely, but besides monkey-patch-chaining I think some
>>>>>>>> modifications would be required, but they would be reasonably simple.
>>>>>>>> Most of the functionality would be handled in one function, which most
>>>>>>>> ufuncs (the ones you care about, as well as ufunc (methods) like add)
>>>>>>>> call. E.g. if ((result = NPy_LazyEval("add", op1, op2)) return result;
>>>>>>>> , which is inserted after argument unpacking and sanity checking. You
>>>>>>>> could also do a per-module hook, and have the function look at
>>>>>>>> sys._getframe(1).f_globals, but that is fragile and won't work from C
>>>>>>>> or Cython code.
>>>>>>>> How did you have overrides in mind?
>>>>>>> My vague idea is that core numpy operations are about as fundamental
>>>>>>> for scientific users as the Python builtin operations are, so they
>>>>>>> should probably be overrideable in a similar way. So we'd teach numpy
>>>>>>> functions to check for methods named like "__numpy_ufunc__" or
>>>>>>> "__numpy_dot__" and let themselves be overridden if found. Like how
>>>>>>> __gt__ and __add__ and stuff work. Or something along those lines.
>>>>>>>> I also found this thread:
>>>>>>>> http://mail.scipy.org/pipermail/numpy-discussion/2011-June/056945.html
>>>>>>>> , but I think you want more than just to override ufuncs, you want
>>>>>>>> numpy to govern when stuff is allowed to be lazy and when stuff should
>>>>>>>> be evaluated (e.g. when it is indexed, slice assigned (although that
>>>>>>>> itself may also be lazy), etc). You don't want some funny object back
>>>>>>>> that doesn't work with things which are not overridden in numpy.
>>>>>>> My point is that probably numpy should *not* govern the decision about
>>>>>>> what stuff should be lazy and what should be evaluated; that should be
>>>>>>> governed by some combination of the user and
>>>>>>> Numba/Theano/minivect/whatever. The toy API I sketched out would make
>>>>>>> those decisions obvious and explicit. (And if the funny objects had an
>>>>>>> __array_interface__ attribute that automatically forced evaluation
>>>>>>> when accessed, then they'd work fine with code that was expecting an
>>>>>>> array, or if they were assigned to a "real" ndarray, etc.)
>>>>>> That's disappointing though, since the performance drawbacks can
>>>>>> severely limit the usefulness for people with big data sets. Ideally,
>>>>>> you would take your intuitive numpy code, and make it go fast, without
>>>>>> jumping through hoops. Numpypy has lazy evaluation,  I don't know how
>>>>>> good a job it does, but it does mean you can finally get fast numpy
>>>>>> code in an intuitive way (and even run it on a GPU if that is possible
>>>>>> and beneficial).
>>>>> All of these proposals require the user to jump through hoops -- the
>>>>> deferred-ufunc NEP has the extra 'with deferredstate' thing, and more
>>>>> importantly, a set of rules that people have to learn and keep in mind
>>>>> for which numpy operations are affected, which ones aren't, which
>>>>> operations can't be performed while deferredstate is True, etc. So
>>>>> this has two problems: (1) these rules are opaque, (2) it's far from
>>>>> clear what the rules should be.
>>>> Right, I guess I should have commented on that. I don't think the
>>>> deferredstate stuff is needed at all, execution can always be deferred
>>>> as long as it does not affect semantics. So if something is marked
>>>> readonly because it is used in an expression and then written to, you
>>>> evaluate the expression and then perform the write. The only way to
>>>> break stuff, I think, would be to use pointers through the buffer
>>>> interface or PyArray_DATA and not respect the sudden readonly
>>>> property. A deferred expression is only evaluated once in any valid
>>>> GIL-holding context (so it shouldn't break threads either).
>>> I think Nathaniel's point is that the point where you get a 10-second
>>> pause to wait for computation is part of the semantics of current NumPy:
>>> print 'Starting computation'
>>> z = (x + y).sum()
>>> print 'Computation done'
>>> print 'Result was', z
>>> I think that if this wasn't the case, newbies would be be tripped up a
>>> lot and things would feel a lot less intuitive. Certainly when working
>>> from the IPython command line.
>>> Also, to remain sane in IPython (or when using a debugger, etc.), I'd want
>>> "print z"
>>> to print something like "unevaluated array", not to trigger a
>>> computation. Same with str(z) and so on.
>> I guess you could detect that at runtime, or just make it
>> configurable. As for triggering computation somewhere else, I guess I
>> find it preferable to horrible performance :)
> My problem might be that I don't use NumPy wherever I need performance
> (except as a glorified double*, i.e. I don't use it for computation).
> NumPy is for interactive "play with a reduced dataset" work.
>>> I don't think a context manager modifying thread-local global state like
>>> with np.lazy:
>>>      ...
>>> would be horribly intrusive.
>>> But I also think it'd be good to start with being very explicit (x =
>>> np.lazy_multiply(a, b); compute(x)) -- such an API should be available
>>> anyway -- and then have the discussion once that works.
>> Maybe that's the best way forward. I guess I'd prefer an import
>> numpy.lazy_numpy as numpy in that case. I don't really like the with
>> statement here, since ideally you'd just experiment with swapping in
>> another module and see if your code still runs fine.
> Or just "import lazyarray as np". As I said, I think it's important to
> refactor NumPy so that things can happen outside of the NumPy project.
> NumPy needs to be very conservative. You've seen the recent NA semantics
> debate. If NumPy was to decide on *the* final blessed semantics for lazy
> evaluation, even as an experimental sub-module, you'd never see the end
> of it.
> One part of this is a polymorphic C API targeted for lazy evaluation and
> get current NumPy to support that. Another part is, as Nathaniel has
> commented, making things like "np.dot" have some kind of polymorphic
> dispatch-on-the-objects behaviour.
> (I'd like something based on multiple dispatch rather than just calling
> something on the left operand. Then use that multiple dispatch for
> implementing +, - and so on as well when you want anything to interact
> with NumPy arrays.)
> Dag
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion


More information about the NumPy-Discussion mailing list