[Numpy-discussion] automatic differentiation with PyAutoDiff

[Nathaniel Smith]

On Thu, Jun 14, 2012 at 3:42 PM, Olivier Grisel
<olivier.grisel at ensta.org> wrote:
> 2012/6/14 James Bergstra <bergstrj at iro.umontreal.ca>:
>> On Thu, Jun 14, 2012 at 4:00 AM, Olivier Grisel
>> <olivier.grisel at ensta.org> wrote:
>>> 2012/6/13 James Bergstra <bergstrj at iro.umontreal.ca>:
>>>> Further to the recent discussion on lazy evaluation & numba, I moved
>>>> what I was doing into a new project:
>>>>
>>>> PyAutoDiff:
>>>> https://github.com/jaberg/pyautodiff
>>>>
>>>> It currently works by executing CPython bytecode with a numpy-aware
>>>> engine that builds a symbolic expression graph with Theano... so you
>>>> can do for example:
>>>>
>>>>>>> import autodiff, numpy as np
>>>>>>> autodiff.fmin_l_bfgs_b(lambda x: (x + 1) ** 2, [np.zeros(())])
>>>>
>>>> ... and you'll see `[array(-1.0)]` printed out.
>>>>
>>>> In the future, I think it should be able to export the
>>>> gradient-computing function as bytecode, which could then be optimized
>>>> by e.g. numba or a theano bytecode front-end. For now it just compiles
>>>> and runs the Theano graph that it built.
>>>>
>>>> It's still pretty rough (you'll see if you look at the code!) but I'm
>>>> excited about it.
>>>
>>> Very interesting. Would it be possible to use bytecode introspection
>>> to printout the compute and display a symbolic representation of an
>>> arbitrary python + numpy expression?
>>>
>>> E.g. something along the lines of:
>>>
>>>>>> g = autodiff.gradient(lambda x: (x + 1) ** 2, [np.zeros(())])
>>>>>> print g
>>> f(x) = 2 * x + 2
>>>>>> g(np.arrange(3))
>>> array[2, 4, 6]
>>>
>>> --
>>> Olivier
>>> http://twitter.com/ogrisel - http://github.com/ogrisel
>>
>> So... almost?
>>
>> I just hacked this gradient function to see what theano could print
>> out, and the first thing that happened (after my own mistakes were
>> sorted out) was an error because the lambda expression was defined to
>> work on a 0-d array, but then you evaluated g on a vector. Was this
>> part of the test? If so, I'm not sure I think it's a good idea, I'm
>> assuming it was a cut-and-paste oversight and moving on....
>
> Indeed, my bad. I wrote that email in a hurry while waiting in line
> for boarding in a plane while still using the airport wifi...
>
>> I settled on (https://github.com/jaberg/pyautodiff/blob/master/autodiff/tests/test_gradient.py)
>> ```
>> import numpy as np
>> from autodiff import Gradient
>>
>> def test_basic():
>>    g = Gradient(lambda x: ((x + 1) ** 2).sum(), [np.zeros(3)])
>>    print g
>>    print g(np.arange(3))
>> ```
>>
>> The output is ... well... ugly but correct:
>> Elemwise{Composite{[mul(i0, add(i1, i2))]}}(TensorConstant{(1,) of
>> 2.0}, TensorConstant{(1,) of 1.0}, <TensorType(float64, vector)>)
>> [array([ 2.,  4.,  6.])]
>
> Indeed it's a bit hard to parse by a human :)
>
>> So with some effort on pretty-printing I'm pretty confident that this
>> could work, at least for simple examples. Pretty-printing is always a
>> challenge for non-trivial examples.  One option might be to convert
>> the internal symbolic graph to sympy?
>
> Indeed that would be great as sympy already has already excellent math
> expression rendering.
>
> An alternative would be to output mathml or something similar that
> could be understood by the mathjax rendering module of the IPython
> notebook.

I'd find it quite useful if it could spit out the derivative as Python
code that I could check and integrate into my source. I often have a
particular function that I need to optimize in many different
situations, but would rather not pull in a whole (complex and perhaps
fragile) bytecode introspection library just to repeatedly recompute
the same function on every run...

-N