[Python-Dev] Release of astoptimizer 0.3

[Victor Stinner]

Hi,

Here are some progress on my astoptimizer project. If you are interested by
the optimizer, run it on your own project or help me to implement more
optimizations.

http://pypi.python.org/pypi/astoptimizer
https://bitbucket.org/haypo/astoptimizer

---

The last version (0.3) works on Python 2.6-3.3 and implements the
following optimizations:

* Call builtin functions if arguments are constants. Examples:

  - len("abc") => 3
  - ord("A") => 65

* Call methods of builtin types if the object and arguments are constants.
  Examples:

  - u"h\\xe9ho".encode("utf-8") => b"h\\xc3\\xa9ho"
  - "python2.7".startswith("python") => True
  - (32).bit_length() => 6
  - float.fromhex("0x1.8p+0") => 1.5

* Call functions of math and string modules for functions without
  border effect. Examples:

  - math.log(32) / math.log(2) => 5.0
  - string.atoi("5") => 5

* Format strings for str%args and print(arg1, arg2, ...) if arguments
  are constants and the format string is valid.
  Examples:

  - "x=%s" % 5 => "x=5"
  - print(1.5) => print("1.5")

* Simplify expressions. Examples:

  - not(x in y) => x not in y
  - 4 and 5 and x and 6 => x and 6

* Loop: replace range() with xrange() on Python 2, and list with
  tuple.  Examples:

  - for x in range(n): ... => for x in xrange(n): ...
  - for x in [1, 2, 3]: ... => for x in (1, 2, 3): ...

* Evaluate unary and binary operators, subscript and comparaison if all
  arguments are constants. Examples:

  - 1 + 2 * 3 => 7
  - not True => False
  - "abc" * 3 => "abcabcabc"
  - abcdef[:3] => abc
  - (2, 7, 3)[1] => 7
  - frozenset("ab") | frozenset("bc") => frozenset("abc")
  - None is None => True
  - "2" in "python2.7" => True
  - "def f(): return 2 if 4 < 5 else 3" => "def f(): return 2"

* Remove dead code. Examples:

  - def f(): return 1; return 2 => def f(): return 1
  - if DEBUG: print("debug") => pass with DEBUG declared as False
  - while 0: ... => pass

---

Unsafe optimizations are disabled by default. Optimizations can be enabled
using a Config class with "features" like "builtin_funcs" (builtin functions
like len()) or "pythonbin" (optimized code will be execute by the same
Python binary executable).

astoptimizer.patch_compile() can be used to hook the optimizer in the
compile() builtin function. On Python 3.3, it is enough to use the optimizer
on imports (thanks to the importlib). On older versions, the compileall
module can be used to compile a whole project using the optimizer.

I didn't start to benchmark anything yet, I focused on fixing bugs (not
generating invalid code). I will start benchmarks when the "variables"
feature (ex: "x=1; print(x)" => "x=1; print(1)") will work. There is an
experimental support of variables, but it is still too agressive and
generate invalid code in some cases (see the TODO file).

I plan to implement other optimizations like unrolling loop or convert
a loop to a list comprehension, see the TODO file.

Don't hesitate to propose more optimizations if you have some ideas ;-)

Victor