Python parser that records source ranges

[Jonathan Edwards]

So the basic idea is to match up the leaves of the AST with the list of 
tokens from tokenizer, which do contain location info. I had thought of 
that, but was hoping there was a more informative parser out there.
Thanks.

Jonathan


logistix at cathoderaymission.net wrote:

> Jonathan Edwards <edwards at nospam.lcs.mit.edu> wrote in message news:<qRKdb.456249$Oz4.260848 at rwcrnsc54>...
> 
>>The parser library module only records source line numbers for tokens. I 
>>need a parser that records ranges of line and character locations for 
>>each AST node, so I can map back to the source. Does anyone know of such 
>>a thing? Thanks
>>
>>Jonathan
> 
> 
> You know there's not going to be a one-to-one relationship, right? 
> Most ast nodes are symbols and aren't going to match to any tokens. 
> Python asts also use a lot of intermediate nodes to enforce operator
> precidence.
> 
> Anyway, I have some rather specialized code in PyXR that syncs tokens
> to an ast.  You probably won't be able to use it out of the box but it
> should give you a good start:
> 
> http://www.cathoderaymission.net/~logistix/PyXR/
> 
> The source file of particular interest to you would be astToHtml.py:
> 
> http://tinyurl.com/p3cn