[Python-Dev] Policy for making changes to the AST

Eugene Toder eltoder at gmail.com
Sun Apr 3 03:55:45 CEST 2011


Hello,

While working on a rewrite of peephole optimizer that works on AST
(http://bugs.python.org/issue11549) I had to make a few changes to the
structure of AST. The changes are described in the issue. This raises the
question -- what is the policy for making changes to the AST? Documentation
for ast module does not warn about possible changes, but obviously changes
can occur, for example, when new constructs are introduced. What about other
changes? Is there a policy for what's acceptable and how this should be
handled?

Assuming we do want to make changes (not just extensions for new language
features), I see 3 options for handling them:

1. Do nothing. This will break code that currently uses AST, but doesn't add
any complexity to cpython.

2. Write a pure-Python compatibility layer. This will convert AST between old
and new formats, so that old code continues working. To do this
a) Introduce ast.compile function (similar to ast.parse), which should be the
recommended way of compiling to AST.
b) Add ast_version argument to ast.parse and ast.compile defaulting to 1.
c) Preserve old AST node classes and attributes in Python.
d) If ast_version specified is not the latest, ast.parse and ast.compile would
convert from/to latest version in Python using ast.NodeTransformer.

This is not fully backward compatible, but allows to do all staging in Python.

3. Full backward compatibility (with Python code). This means conversion is
done in compile(). It can either call Python conversion code from ast module,
or actually implement conversion in C using AST visitors. Using my visitors
generator this should not be very hard. Downsides here are a lot of C code and
no clear separation of deprecated AST nodes (they will remain in Python.asdl).
Otherwise it's similar to 2, with ast_version argument added to compile() and
ast.parse.

For 2 and 3 we can add a PendingDeprecationWarning when ast_version 1 is used.

In any case, C extensions that manipulate AST will be broken, but 3 provides a
simple way to update them -- insert calls to C conversion functions.

Eugene


More information about the Python-Dev mailing list