[Python-Dev] astoptimizer: static optimizer working on the AST

Victor Stinner victor.stinner at gmail.com
Tue Mar 26 22:56:53 CET 2013


I made progress since last August on my astoptimizer project (read the
Changelog). Previous email thread:

The astoptimizer project is an optimizer rewriting Python AST. It
executes as much code as possible during the compilation. The
optimizer itself is not designed to be fast, but to emit faster code.


Some optimizations are not "pythonic": don't respect the Python
language by making some assumptions (ex: on the namespace), which may
be wrong in some cases. astoptimizer is written for adults which know
which optimizations can be enabled, and which ones must be disabled in
their application (application, not module). I'm trying to write a
safe and "pythonic" default configuration. For example, len("abc") is
not replaced with 3 by default. You have to enable explicitly the
"builtin_funcs" configuration feature.

astoptimizer can be used as a pythonic preprocessor: it replaces
os.name, sys.platform and also your own constants by their value. It
removes dead code and so may be used to remove completly the overhead
of checks on the Python version or an the platform (if python3: ...
else: ...).

I would like to improve the integration of astoptimizer in Python 3.4.
Brett Canon asked me to use an hook in importlib, I proposed to add a
generic AST hook which would also be called by eval(), compile(), etc.
I would like to allow anyone to use its own AST modifier, and simplify
the usage of astoptimizer.
=> http://bugs.python.org/issue17515

Open issues:

 * what should be the name of "pyc" files?
 * how to handle different configuration of astoptimizer: generate
different "pyc" files?
 * no sys.getasthook(): it would permit to chain different AST hooks


There are many open issues proposing to write a better Python
optimizer. Some of them are stuck because implementing them using the
bytecode optimizer is hard.

 * http://bugs.python.org/issue1346238:
   A constant folding optimization pass for the AST
 * http://bugs.python.org/issue2181:
   optimize out local variables at end of function
 * http://bugs.python.org/issue2499:
   Fold unary + and not on constants
 * http://bugs.python.org/issue4264:
   Patch: optimize code to use LIST_APPEND instead of calling list.append
 * http://bugs.python.org/issue7682:
   Optimisation of if with constant expression
 * http://bugs.python.org/issue10399:
   AST Optimization: inlining of function calls
 * http://bugs.python.org/issue11549:
   Build-out an AST optimizer, moving some functionality out of the
peephole optimizer
 * http://bugs.python.org/issue17068:
   peephole optimization for constant strings
 * http://bugs.python.org/issue17430:
   missed peephole optimization

astoptimizer implements many optimizations listed in these issues.
Read the README file to the list of optimizations already implemented,
and the TODO file for ideas of new optimizations.


The project is still experimental. I ran Python 2.7 and 3.4 test
suites. Some tests are failing because the AST or the bytecode is
different (which is expected). More tests are failing if you enable
more agressive optimizations (especially if you enable the agressive
mode to remove dead code). At least, Python does not crash :-) (It
occurs sometimes when astoptimizer generates invalid AST!)


More information about the Python-Dev mailing list