[Python-Dev] AST optimizer implemented in Python

[Gregory P. Smith]

On Sat, Aug 11, 2012 at 11:30 AM, Victor Stinner
<victor.stinner at gmail.com>wrote:

> Hi,
>
> I started to implement an AST optimizer in Python. It's easy to create
> a new AST tree, so I'm surprised that I didn't find any existing
> project.
>
> https://bitbucket.org/haypo/misc/src/tip/python/ast_optimizer.py


Neat!


>
>
> To test its peephole optimizations (by checking manually its final
> bytecode), I wrote a patch for Python to disable Python internal
> peephole optimizer (on bytecode):
>
> https://bitbucket.org/haypo/misc/src/tip/python/compile_disable_peephole.patch
>
> --
>
> There is BytecodeAssembler [1], but it seems to be specialized on
> bytecode. There are (at least?) 3 different issues to implement an AST
> optimizer, but in C, not in Python:
>
> http://bugs.python.org/issue1346238
> http://bugs.python.org/issue10399
> http://bugs.python.org/issue11549
>
> --
>
> My proof-of-concept only implements very basic optimizations like 1+1
> => 2 or "abcdef"[:3] => "abc", but it should easy to extend it to do
> more interesting optimization like function inlining.
>
> To allow more optimization, the optimizer permits to declare variables
> as constant. For example, sys.hexversion is replaced by its value by
> the optimizer, and checks like "sys.hexversion > 0x3000000" are done
> at compile time. This feature can be used to drop completly debug code
> at compilation, without the need of a preprocessor. For example, calls
> to logging.debug() can simply be dropped if the log level is hardcoded
> to ERROR (40).
>
> Other idea to improve this optimizer:
>  - move invariant out of loops. Example: "x=[]; for i in range(10):
> x.append(i)" => "x=[]; x_append=x.append; for i in range(10):
> x_append(i)". Require to infer the type of variables.
>  - unroll loops
>  - inline small functions
>  - etc.
>
> To be able to use such optimizer on a whole Python project, a new
> function (ex: sys.setastoptimizer()) can be added to CPython to call
> the optimizer between the compilation to AST and the compilation to
> bytecode. So the optimizer would be optionnal and it avoids bootstrap
> issues.
>

+1  We should add some form of setastoptimizer API in 3.4.  Please start a
PEP for this.  It would be nice to include the ability to properly cache
the ast optimizer output so that it does not have to run every time (in pyc
files or similar, etc) but can verify that it is the specific ast optimizer
that ran on a give existing cached copy.

-gps
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20120811/1e66e72a/attachment.html>