Parsing and Editing Source

[eliben]

On Aug 15, 4:21 pm, "Paul Wilson" <paulalexwil... at gmail.com> wrote:
> Hi all,
>
> I'd like to be able to do the following to a python source file
> programmatically:
>  * Read in a source file
>  * Add/Remove/Edit Classes, methods, functions
>  * Add/Remove/Edit Decorators
>  * List the Classes
>  * List the imported modules
>  * List the functions
>  * List methods of classes
>
> And then save out the result back to the original file (or elsewhere).
>
> I've begun by using the tokenize module to generate a token-tuple list
> and am building datastructures around it that enable the above
> methods. I'm find that I'm getting a little caught up in the details
> and thought I'd step back and ask if there's a more elegant way to
> approach this, or if anyone knows a library that could assist.
>
> So far, I've got code that generates a line number to token-tuple list
> dictionary, and am working on a datastructure describing where the
> classes begin and end, indexed by their name, such that they can be
> later modified.
>
> Any thoughts?
> Thanks,
> Paul

Consider using the 'compiler' module which will lend you more help
than 'tokenize'.

For example, the following demo lists all the method names in a file:

import compiler

class MethodFinder:
    """ Print the names of all the methods

        Each visit method takes two arguments, the node and its
        current scope.
        The scope is the name of the current class or None.
    """

    def visitClass(self, node, scope=None):
        self.visit(node.code, node.name)

    def visitFunction(self, node, scope=None):
        if scope is not None:
            print "%s.%s" % (scope, node.name)
        self.visit(node.code, None)

def main(files):
    mf = MethodFinder()
    for file in files:
        f = open(file)
        buf = f.read()
        f.close()
        ast = compiler.parse(buf)
        compiler.walk(ast, mf)

if __name__ == "__main__":
    import pprint
    import sys

    main(sys.argv)