[Python-Dev] AST optimizer implemented in Python

[Victor Stinner]

> I started to implement an AST optimizer in Python. It's easy to create
> a new AST tree, so I'm surprised that I didn't find any existing
> project.

I done more research. I found the AST optimizer of PyPy, which
implements basic optimizations:
https://bitbucket.org/pypy/pypy/src/default/pypy/interpreter/astcompiler/optimize.py

So there is also something like an AST optimizer in Cython:
https://github.com/cython/cython/blob/master/Cython/Compiler/Optimize.py
https://github.com/cython/cython/blob/master/Cython/Compiler/ParseTreeTransforms.py
https://github.com/cython/cython/blob/master/Cython/Compiler/Builtin.py
https://github.com/cython/cython/blob/master/Cython/Compiler/Pipeline.py#L123

--

> https://bitbucket.org/haypo/misc/src/tip/python/ast_optimizer.py

I moved the script to a new dedicated project on Bitbucket:
https://bitbucket.org/haypo/astoptimizer

Join the project if you want to help me to build a better optimizer!

It now works on Python 2.5-3.3.

> There is BytecodeAssembler [1], but it seems to be specialized on
> bytecode. There are (at least?) 3 different issues to implement an AST
> optimizer, but in C, not in Python:
>
> http://bugs.python.org/issue1346238
> http://bugs.python.org/issue10399
> http://bugs.python.org/issue11549

Oh, http://bugs.python.org/issue10399 includes an optimizer
implemented in Python: Lib/__optimizer__.py. It inlines functions and
create specialized versions of a function.

> My proof-of-concept only implements very basic optimizations like 1+1
> => 2 or "abcdef"[:3] => "abc", but it should easy to extend it to do
> more interesting optimization like function inlining.

It is also possible to call functions and methods at compile time, if
there have no border effect (and don't depend on the environnement).
For example, len("abc") is always 3. I added a lot of such functions,
especially builtin functions.

I added options to enable more aggressive optimizations if we know
that result will run on the same host than the compiler (os.name,
sys.byteorder, ... are replaced by their value).

Victor