Hi Laurent
Great to see somebody finally tackling refactoring.
I'm answering, because I think we're working on the same issue. But we have
finished two different parts: You have finished a refactoring
implementation and I have finished the static analysis part. I'm the author
of Jedi. https://github.com/davidhalter/jedi/
I'm currently working on the integration of the lib2to3 parser into Jedi.
This would make refactoring really easy (I'm about 50% done with the
parser). It's also well tested and offers a few other advantages.
In a perfect world, we could now combine our projects :-) I will look in
detail at Red Baron on Monday.
~ Dave
PS: If you want to use a tool for static analysis, please use either Jedi
or astroid, I don't think rope still makes sense, because the project is
inactive.
2014-11-14 13:05 GMT+01:00 Laurent Peuch
Hello everyone,
Someone has suggested me to talk about the project I'm working on right now on this mailing list because this has a lot of chances to interest you.
This tool is an answer to a frustration that I've had while trying to build tools for python and projects I was working on. While there is already good capacities in python to analyse code (ast.py, astroid (while it wasn't out at that time)), the "write code that modify source code" was really missing (in my opinion and my knowledge of the existing tools).
I wanted a very intuitive and easy to use library that allows me to query and modify my source code only in the place I wanted to modify it without touching the rest of the code. So I've built what can be describe as "the BeautifulSoup of python source code".
To do so, I've built what can be called "a lossless AST" for python (designed to be used by humans), an AST that satisfy this equation:
source_code == ast_to_source(source_to_ast(source_code))
It produces json-serializable python data structures (because data structures are easier to use and don't hide anything from you).
And now the part that should interest you more: on top on that AST, I've built an high level "query and modification" library that wraps those AST nodes into objects. I've put a lot of efforts in making this library intuitive and very easy to use while removing you the burden of having to deal with low level details. This "BeautifulSoup of the python source code" is called Redbaron.
It looks like this:
from redbaron import RedBaron
# simple API
# pass string red = RedBaron("some_value = 42")
# get string back red.dumps()
Queries are like BeautifulSoup:
red.find("int", value=4) red.find_all("def", name="stuff")
(You can pass lambda/regex/special syntaxe for globs/regex etc... to queries, they should be powerful enough for the vast majorities of your needs).
Nodes modification is very simple: just pass source code stored in string and "voilà":
red = RedBaron("some_value = 42") red[0].value = "1 + 1" # some_value = 1 + 1
red = RedBaron("def stuff():\n plop") red[0].value = "some_code" # def stuff():\n some_code
# notice that the input is correctly formatting, indented and it # also takes care of not breaking the next node indentation # works too with decorators and where you expect it to works
(It is possible to pass it ast datastructure or RedBaron objects to).
And I've made an abstraction on top of "list of things" so you don't have to take care about when you need to put a separator (for eg: a "," in a list):
red = RedBaron("[1, 2, 3]") red[0].append("plop") # [1, 2, 3, plop]
# want to add a new django app to INSTALLED_APPS? just do: red.find("assignment", target=lambda x: x.dumps() == "INSTALLED_APPLICATIONS").value.append("'another_app'") # notice that the formatting of the list is detected
# want to add "@profile" to every function of the root level for # line_profiler? red('def', recursive=False).map(lambda x: x.decorators.insert(0, '@profile'))
# and remove them red("decorator", lambda x: x.dumps() == "@decorator").map(lambda x: x.parent.parent.decorators.remove(x))
# convert every "print a" to "logger.debug(a) red('print', value=lambda x: len(x) == 1).map(lambda x: x.replace('logger.debug(%s)' % x.value.dumps())
# and print a, b, c to logger.debug("%s %s %s" % (a, b, c)) red('print', value=lambda x: len(x) == 1).map(lambda x: x.replace('logger.debug("%s" % (%s))' % (" ".join('%s' * len(x.value)))
Both library and fully tested (more than 2000 tests in total), fully *documented* (with lots of examples) and under freesoftware licences. I consider RedBaron to be in alpha stage, it is already very stable but a significant number of edge cases are probably not handled yet.
Important point: RedBaron is not and will not do static analysis, I'm probably going to integrate (or integrate RedBaron into) a tool that already do that like astroid or rope.
Links:
* RedBaron tutorial: https://redbaron.readthedocs.org/en/latest/tuto.html * RedBaron documentation: https://redbaron.readthedocs.org * RedBaron source code: https://github.com/psycojoker/redbaron
* Baron (the AST) source code: https://github.com/psycojoker/baron * Baron documentation: https://baron.readthedocs.org
I hope that I have trigger your interest and I'm very interested by your feedback,
Have a nice day and thanks for your time,
PS: I've only been aware of the capacities of lib2to3 since 2 months and was very unhappy to discover it so late (I've spent months or googling before deciding to start this project), I'll probably swap my parser with lib2to3 one in the future.
--
Laurent Peuch -- Bram _______________________________________________ code-quality mailing list code-quality@python.org https://mail.python.org/mailman/listinfo/code-quality