[Python-Dev] ast-objects branch created

[Brett Cannon]

On 12/5/05, Neal Norwitz <nnorwitz at gmail.com> wrote:
> On 12/5/05, Jeremy Hylton <jeremy at alum.mit.edu> wrote:
[SNIP]
> I didn't look at the changes much, but from what I saw I think it may
> be better to move the arenas off the branch and onto the head now.  It
> appears to be much easier to get right since there is virtually no
> error handling code in line.  It's all taken care of in a few central
> places.
>
> We can then decide between the arenas in the head vs PyObjects.
>

I am also +1 with merging the arena into the trunk.  The arena
approach compared to the existing solution is a lot easier to use. 
With almost all calls to the arena in the auto-generated constructor
code, one just has to make sure that key places have PyArena_Free() to
free the arena and that errors propagate up to those points.

But, as Neal is suggesting, this should not prevent the PyObject
version from moving forward since it could still turn out to be the
better solution.

> > > I'd also like to parrot the concern others have had that if the AST
> > > nodes are not made of PyObjects, then a mirror hierarchy of PyObject-
> > > ified AST nodes will have to be created, which seems like quite a
> > > wasteful duplication. If it is required that there be a collection of
> > > AST python objects (which I think it is), is there really any good
> > > reason to make the "real" AST objects not be the _only_ AST objects?
> > > I've not heard one.
> >
> > The PyObject-ified AST nodes are only needed if user code requests an
> > AST from the compiler.  That is, if we add a new feature that exposes
> > AST, we would need AST objects represented in Python code.  I think
> > this feature would be great to add, but it doesn't seem like a primary
> > concern for the internal compiler implementation.
>
> FWIW, I agree with this approach.  I don't care that much about the
> internal AST for its own sake.  I want to consume the AST and I only
> care about the internals insofar as the result is correct and
> maintainable.
>

It really comes down to how people expect to use the exposure of the
AST.  If we try to make sure there is no horrible overhead in getting
the AST to Python code and then to the bytecode compiler then it can
be used for optimizations (e.g., the existing peepholer could be
rewritten in Python and just a default transformation that the AST is
passed through).

But if we don't want to make sure that AST access is used for
optimization transformation but more for non-performance critical uses
(e.g., error checking ala PyChecker or refactoring tools) then the
simplest, easiest to maintain solution should win out.

Personally I want the former abilities for academic experimentation
reasons.  I don't think that a bunch of optimizations are suddenly
going to appear out of nowhere for Python code, but I still would like
to be able to experiment with some without having to worry about a
performance penalty for doing so.

Granted, though, if we byte-compiled scripts passed in on the
command-line we would definitely help minimize the performance impact.
 Interpreter input might be a little slower, but then again since it
will be such bite-sized chunks of AST a couple more Python calls
shouldn't be that significant.

Plus I don't know if serialization will be that much slower than
passing the AST itself out since doing a full transformation on an AST
might be extremely more costly than just getting the AST to the Python
code in the first place.

> So my view of the best approach is one that is easy to get right and
> maintain.  That's why I think the arena should be moved to the head
> now.  From what I saw it was much easier to get right, it removed a
> bunch of code and should be more maintainable.
>
> I will also probably work on the PyObject approach, since if that's
> more maintainable I'd prefer that in the end.  I don't know which
> approach is best.
>
> I also really like Martin's idea about generating a lot more (all?) of
> the manually written Python/ast.c code.  I'd prefer much less C code
> to maintain.
>

A new sprint topic for PyCon for Guido to give us a month deadline on
after we have worked on it for three years!  =)

-Brett