[Python-Dev] Fwd: Re: ANN compiler2 : Produce bytecode from Python 2.5 AST

[Michael Spencer]

Martin v. Löwis wrote:
> Georg Brandl schrieb:
>> Perhaps you can bring up a discussion on python-dev about your improvements
>> and how they could be integrated into the standard library...
> 
> Let me second this. The compiler package is largely unmaintained and
> was known to be broken (and perhaps still is). A replacement
> implementation, especially if it comes with a new maintainer, would
> be welcome.
> 
> Regards,
> Martin

Hello python-dev.

I use AST-based code inspection and manipulation, and I've been looking forward 
to using v2.5 ASTs for their increased accuracy, consistency and speed. However, 
there is as yet no Python-exposed mechanism for compiling v2.5 ASTs to bytecode.

So to meet my own need and interest I've been implementing 'compiler2', similar 
in scope to the stblib compiler package, but generating code from Python 2.5 
_ast.ASTs.  The code has evolved considerably from the compiler package: in 
aggregate the changes amount to a re-write.  More about the package and its 
status below.

I'm introducing this project here to discuss whether and how these changes 
should be integrated with the stdlib.

I believe there is a prima facie need to have a builtin/stdlib capability for 
compiling v2.5 ASTs from Python, and there is some advantage to having that be 
implemented in Python.  There is also a case for deprecating the v2.4 ASTs to 
ease maintenance and reduce the confusion associated with two different AST formats.

If there is interest, I'm willing make compiler2 stdlib-ready.  I'm also open to 
alternative approaches, including doing nothing.

compiler2 Objectives and Status
===============================
My goal is to get compiler2 to produce identical output to __builtin__.compile 
(at least optionally), while also providing an accessible framework for 
AST-manipulation, experimental compiler optimizations and customization.

compiler2 is not finished - there are some unresolved bugs, and open questions 
on interface design - but already it produces identical output to 
__builtin__.compile for all of the stdlib modules and their tests (except for 
the stackdepth attribute which is different in 12 cases). All but three stdlib 
modules pass their tests after being compiled using compiler2.  More on goals, 
status, known issues etc... in the project readme.txt at: 
http://svn.brownspencer.com/pycompiler/branches/new_ast/readme.txt

Code is available in Subversion at 
http://svn.brownspencer.com/pycompiler/branches/new_ast/

The main test script is test/test_compiler.py which compiles all the modules in 
/Lib and /Lib/test and compares the output with __builtin__.compile.


Best regards

Michael Spencer