<div dir="ltr">Hello Dave,<div><br></div><div>in my Python compiler Nuitka, I am using the "ast" module, and it's working fine for many things, but it lends itself badly to other things. I am condemned to do my own static analysis, as it's intended for optimization.</div><div><br></div><div>My interest is in the reporting and also auto-format of source code. </div><div><br></div><div>I would love to be able to report about source code easily for "profile guided optimization", or make annotations about Nuitka's finding in HTML reports from generated output.</div><div><br></div><div>And I want a coherent code base for my private and work projects, and would like to be able to apply formatting, even function call style changes automatically, but on a programmatic base.</div><div><br></div><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">I think the two tools are very similar. The biggest difference is<br></blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
probably static analysis, which you definitely need for certain<br>
refactorings.<br></blockquote><div><br></div><div>Static analysis is great for auto-format indeed. Being e.g. able to tell that a method is only there for overload because it raises a "must be overloaded" exception, that kind of thing would otherwise be too hard.</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">However Jedi definitely has fewer AST functions. The node/leaf objects<br>
of Jedi are at the moment quite simple. I'm willing to add<br>
functionality there, but only if it's used. Currently there's only the<br>
functions there that Jedi needs internally. To support the well known<br>
refactorings (e.g. inline/extract name/function), we might need to add<br>
a few methods there.<br></blockquote><div><br></div><div>With RedBaron, I can do this (bare with me on code quality, and by no means assume it's competent RedBaron usage):</div><div><br></div><div><div>def updateString(string_node):</div><div> # Skip doc strings for now.</div><div> if string_node.parent.type in ("class", "def", None):</div><div> return</div><div><br></div><div> value = string_node.value</div><div><br></div><div> def isQuotedWith(quote):</div><div> return value.startswith(quote) and value.endswith(quote)</div><div><br></div><div> for quote in "'''", '"""', "'", '"':</div><div> if isQuotedWith(quote):</div><div> break</div><div> else:</div><div> assert False, value</div><div><br></div><div> real_value = value[len(quote):-len(quote)]</div><div> assert quote + real_value + quote == value</div><div><br></div><div> if "\n" not in real_value:</div><div> # Single characters, should be quoted with "'"</div><div> if len(eval(value)) == 1:</div><div> if real_value != "'":</div><div> string_node.value = "'" + real_value + "'"</div><div> else:</div><div> if '"' not in real_value:</div><div> string_node.value = '"' + real_value + '"'</div></div><div><br></div><div>And then call this:</div><div><br></div><div><div>for node in red.find_all("StringNode"):</div><div> try:</div><div> updateString(node)</div><div> except Exception:</div><div> print("Problem with", node)</div><div> node.help(deep = True, with_formatting = True)</div><div> raise</div></div><div><br></div><div>It allows me do enforce some rules according to strings that are not multi-line. My rules there are, do not use triple quites without a new-line, strings of resulting length 1, become 'a' or '\n' and others, use "" or "ab", except of course "'" is valid and 'some "quote" would be too' as well. </div><div><br></div><div>I do a bunch of these, and like to have these things, to e.g. make sure that my multi-line calling convention lines up the "=" signs nicely, etc, inserting and removing white space from tuples, comma separated stuff, etc.</div><div><br></div><div>To me, it's not about, if I should do this, but if it can be done.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><span class="">> Does it provide a bounding box for code constructs?<br>
<br>
</span>I'm not really sure what you mean. Jedi knows the exact positions of<br>
objects. At the moment there's no method like RedBaron's<br>
`bounding_box`. Relative positions could be easily calculated with the<br>
current parser. However, I don't know what such a BoundingBox would be<br>
doing.<br></blockquote><div><br></div><div>That is the caret, typically used in stack traces, and fits the concept of a cursor. If I go to a report, then I will want to have the bounding box.</div><div><br></div><div>When e.g. highlighting a function or a call expression, or argument expression, I am not only interested in where it starts, but where it ends. For an XHTML report of Nuitka performance compared to CPython performance on the same code (which is my plan for this autumn, at about the time it starts to make actual sense), with the "ast" module and apparently "jedi", all I get is this:</div><div><br></div><div>f( arg1(), arg2(), c ** d) + g()</div><div><br></div><div>And I would like to mouse over or highlight and know where the call to f() ends, where the third argument ends, what the operation "+" entails. Without a bounding box, that falls down. In fact, I would also want some position, like the "+" or "(" to indicate which bounding box I mean.</div><div><br></div><div>My problem there with the "ast" module boils down to this:</div><div><br></div><div><div>>>> ast.parse("a+b").body[0].value.col_offset</div><div>0</div><div>>>> ast.dump(ast.parse("a+b").body[0])</div><div>"Expr(value=BinOp(left=Name(id='a', ctx=Load()), op=Add(), right=Name(id='b', ctx=Load())))"</div></div><div><br></div><div>With RedBaron, I get to have a "bounding_box". I must admit to not yet have used it. But I aspire to. And even for Nuitka, I would love to have the position of the "+" for use tracebacks in at least improved mode, as opposed to the first argument. But performance and bugs are keeping me away from considering any alternatives to "ast" there.</div><div><br></div><div>So sure, I am asking of Jedi, if it has that bounding box, precisely to address this.</div><div><br></div><div>With RedBaron, I can do this:</div><div><br></div><div><div>>>> from redbaron import RedBaron</div><div>>>> red = RedBaron("a+b")</div></div><div><div>>>> red[0]</div><div>a+b</div></div><div><br></div><div><div>>>> red[0].second_formatting = " "</div><div>>>> red</div><div>0 a+ b</div></div><div><br></div><div>That of course also means, it knows the "+" location, or I can infer it:</div><div><br></div><div><div>>>> red[0].second.bounding_box</div><div>BoundingBox (Position (1, 1), Position (1, 1))</div><div>>>> red[0].first.bounding_box</div><div>BoundingBox (Position (1, 1), Position (1, 1))</div><div>>>> red[0].bounding_box<br></div><div>BoundingBox (Position (1, 1), Position (1, 5))</div></div><div><br></div><div>Seems bounding boxes are relative, but it also has "get_absolute_bounding_box_of_attribute".</div><div><br></div><div>So, eventually I am faced with the issue of producing run time information from expressions, and then to find the same expression again in two different "ast" forms. But for rendering and editing, both RedBaron seems like it might work with heavy fighting.</div><div><br></div><div>I would love for you to provide a code example how to use Jedi for editing like I did above, and if you think that creating such reports could be based on Jedi parsing.</div><div><br></div><div>My plan for Nuitka now entails to probably identify an expression in Nuitka uniquely by path. That "uid" for code in a module is probably a new idea. The "uid" could be a run time number, hash code, which is then resolvable looking at the original code again in a new parse with even another tool that provides more detail.</div><div><br></div><div>Finding it then in RedBaron or Jedi again may involve some normalization. The "ast" module hides some things from me, e.g. try/except/finally is nested statements in at least Python2.</div><div><br></div><div>Yours,</div><div>Kay</div><div><br></div></div></div></div>