[Python-Dev] Fwd: Re: ANN compiler2 : Produce bytecode from Python 2.5 AST
mahs at telcopartners.com
Wed Oct 25 23:06:21 CEST 2006
Martin v. Löwis wrote:
> Georg Brandl schrieb:
>> Perhaps you can bring up a discussion on python-dev about your improvements
>> and how they could be integrated into the standard library...
> Let me second this. The compiler package is largely unmaintained and
> was known to be broken (and perhaps still is). A replacement
> implementation, especially if it comes with a new maintainer, would
> be welcome.
I use AST-based code inspection and manipulation, and I've been looking forward
to using v2.5 ASTs for their increased accuracy, consistency and speed. However,
there is as yet no Python-exposed mechanism for compiling v2.5 ASTs to bytecode.
So to meet my own need and interest I've been implementing 'compiler2', similar
in scope to the stblib compiler package, but generating code from Python 2.5
_ast.ASTs. The code has evolved considerably from the compiler package: in
aggregate the changes amount to a re-write. More about the package and its
I'm introducing this project here to discuss whether and how these changes
should be integrated with the stdlib.
I believe there is a prima facie need to have a builtin/stdlib capability for
compiling v2.5 ASTs from Python, and there is some advantage to having that be
implemented in Python. There is also a case for deprecating the v2.4 ASTs to
ease maintenance and reduce the confusion associated with two different AST formats.
If there is interest, I'm willing make compiler2 stdlib-ready. I'm also open to
alternative approaches, including doing nothing.
compiler2 Objectives and Status
My goal is to get compiler2 to produce identical output to __builtin__.compile
(at least optionally), while also providing an accessible framework for
AST-manipulation, experimental compiler optimizations and customization.
compiler2 is not finished - there are some unresolved bugs, and open questions
on interface design - but already it produces identical output to
__builtin__.compile for all of the stdlib modules and their tests (except for
the stackdepth attribute which is different in 12 cases). All but three stdlib
modules pass their tests after being compiled using compiler2. More on goals,
status, known issues etc... in the project readme.txt at:
Code is available in Subversion at
The main test script is test/test_compiler.py which compiles all the modules in
/Lib and /Lib/test and compares the output with __builtin__.compile.
More information about the Python-Dev