<div dir="ltr">I don't think I'm asking for so much. Somewhere inside numexpr it builds an AST of its own, which it converts into the optimized code. It would be more useful to me if that AST were in the same format as the one returned by Python's ast module. This way, I could glue in the bits of numexpr that I like with my code. For my purpose, this would have been the more ideal design.<div><br></div><div><div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Apr 27, 2015 at 10:47 PM, Nathaniel Smith <span dir="ltr"><<a href="mailto:njs@pobox.com" target="_blank">njs@pobox.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="HOEnZb"><div class="h5"><p dir="ltr">On Apr 27, 2015 5:30 PM, "Neil Girdhar" <<a href="mailto:mistersheik@gmail.com" target="_blank">mistersheik@gmail.com</a>> wrote:<br>
><br>
><br>
><br>
> On Mon, Apr 27, 2015 at 7:42 PM, Nathaniel Smith <<a href="mailto:njs@pobox.com" target="_blank">njs@pobox.com</a>> wrote:<br>
>><br>
>> On Mon, Apr 27, 2015 at 4:23 PM, Neil Girdhar <<a href="mailto:mistersheik@gmail.com" target="_blank">mistersheik@gmail.com</a>> wrote:<br>
>> > I was told that numba did similar ast parsing, but maybe that's not true.<br>
>> > Regarding the ast, I don't know about reliability, but take a look at<br>
>> > get_ast in pyautodiff:<br>
>> > <a href="https://github.com/LowinData/pyautodiff/blob/7973e26f1c233570ed4bb10d08634ec7378e2152/autodiff/context.py" target="_blank">https://github.com/LowinData/pyautodiff/blob/7973e26f1c233570ed4bb10d08634ec7378e2152/autodiff/context.py</a><br>
>> > It looks up the __file__ attribute and passes that through compile to get<br>
>> > the ast. Of course that won't work when you don't have source code (a .pyc<br>
>> > only module, or when else?)<br>
>> ><br>
>> > Since I'm looking into this kind of solution for the future of my code, I'm<br>
>> > curious if you think that's too unreliable for some reason?<br>
>><br>
>> I'd certainly hesitate to rely on it for anything I cared about or<br>
>> would be used by a lot of people... it's just intrinsically pretty<br>
>> hacky. No guarantee that the source code you find via __file__ will<br>
>> match what was used to compile the function, doesn't work when working<br>
>> interactively or from the ipython notebook, etc. Or else you have to<br>
>> trust a decompiler, which is a pretty serious complex chunk of code<br>
>> just to avoid typing quote marks.<br>
><br>
><br>
> Those are all good points. However, it's more than just typing quote marks. The code might have non-numpy things mixed in. It might have context managers and function calls and so on. More comments below.<br>
> <br>
>><br>
>><br>
>> > From a<br>
>> > usability standpoint, I do think that's better than feeding in strings,<br>
>> > which:<br>
>> > * are not syntax highlighted, and<br>
>> > * require porting code from regular numpy expressions to numexpr strings<br>
>> > (applying a decorator is so much easier).<br>
>><br>
>> Yes, but then you have to write a program that knows how to port code<br>
>> from numpy expressions to numexpr strings :-). numexpr only knows a<br>
>> tiny restricted subset of Python...<br>
>><br>
>> The general approach I'd take to solve these kinds of problems would<br>
>> be similar to that used by Theano or dask -- use regular python source<br>
>> code that generates an expression graph in memory. E.g. this could<br>
>> look like<br>
>><br>
>> def do_stuff(arr1, arr2):<br>
>> arr1 = deferred(arr1)<br>
>> arr2 = deferred(arr2)<br>
>> arr3 = np.sum(arr1 + (arr2 ** 2))<br>
>> return force(arr3 / np.sum(arr3))<br>
>><br>
>> -n<br>
>><br>
><br>
> Right, there are three basic approaches: string processing, AST processing, and compile-time expression graphs.<br>
><br>
> The big advantage to AST processing over the other two is that you can write and test your code as regular numpy code along with regular tests. Then, with the application of a decorator, you get the speedup you're looking for. The problem with porting the numpy code to numexpr strings or Theano-like expression-graphs is that porting can introduce bugs, and even if you're careful, every time you make a change to the numpy version of the code, you have port it again.<br>
><br>
> Also, I personally want to do more than just AST transformations of the numpy code. For example, I have some methods that call super. The super calls can be collapsed since the mro is known at compile time.</p>
</div></div><p dir="ltr">If you want something that handles arbitrary python code ('with' etc.), and produces results identical to cpython (so tests are reliable), except in cases where it violates the semantics for speed (super), then yeah, you want a full replacement python implementation, and I agree that the proper input to a python implementation is .py files :-). That's getting a bit far afield from numexpr's goals though...</p><span class="HOEnZb"><font color="#888888">
<p dir="ltr">-n</p>
</font></span><br>_______________________________________________<br>
NumPy-Discussion mailing list<br>
<a href="mailto:NumPy-Discussion@scipy.org">NumPy-Discussion@scipy.org</a><br>
<a href="http://mail.scipy.org/mailman/listinfo/numpy-discussion" target="_blank">http://mail.scipy.org/mailman/listinfo/numpy-discussion</a><br>
<br></blockquote></div><br></div></div></div></div>