Range information in the AST -- once more
Hello! I'm writing a static language analyzer for an IDE which reuses the CPython parser (for parsing) [1]. Two years ago, I asked about a few changes to be made to the AST provided by CPython, but the discussion thread dried up before a definite decision was made. I decided to just copy the parser code to my project and make the necessary changes there back then. I'm bringing this up again now since after my project has seen its first release recently, packagers are (understandably) a bit unhappy about the python fork residing in its repository. I would really like to get rid of that fork and link against the vanilla libpython instead. There's two things which are at the very least required to make this work: 1. The col_offset and lineno of an Attribute must give the beginning of the word that names the attribute, not the beginning of the expression. Example: In "foo.bar.baz", the col_offset of the Attribute belonging to "bar" says "0" currently, it would need to be "4". 2. Column offsets and line numbers would need to be available for function arguments (those with and without stars), and for alias nodes. In total, this requires very little change to the existing code, "a few tens of lines changed at most" order of magnitude; those are mostly trivial changes. For what I can tell, the impact on existing code using the AST stuff will be about zero. Even if there was some really obscure case where the change would matter, porting would only require about three lines of python code. Additionally, there's a few more things which would be useful to have available from the AST (namely the ranges of class and function names when they are defined -- currently only the start of the first decorator is available), but since those are reasonably easy to work around it's not that important. It would still be nice tough. If you think this is a reasonable suggestion then I'll be happy to provide a patch for more detailed discussion. Greetings, Sven ________ [1] See https://projects.kde.org/kdev-python
On Thu, Dec 27, 2012 at 11:03 PM, Sven Brauch <svenbrauch@googlemail.com> wrote:
Hello!
I'm writing a static language analyzer for an IDE which reuses the CPython parser (for parsing) [1]. Two years ago, I asked about a few changes to be made to the AST provided by CPython, but the discussion thread dried up before a definite decision was made. I decided to just copy the parser code to my project and make the necessary changes there back then. I'm bringing this up again now since after my project has seen its first release recently, packagers are (understandably) a bit unhappy about the python fork residing in its repository. I would really like to get rid of that fork and link against the vanilla libpython instead.
It certainly sounds like its worth considering for 3.4. It's a new feature, though, so it unfortunately wouldn't be possible to backport it to any earlier releases. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
2012/12/27 Nick Coghlan <ncoghlan@gmail.com>:
It certainly sounds like its worth considering for 3.4. It's a new feature, though, so it unfortunately wouldn't be possible to backport it to any earlier releases.
Yes, that is understandable. It wouldn't be much of a problem tough, my whole project is pretty bleeding-edge anyways, and depending on python >= 3.4 wouldn't hurt. For me it would only be important to have an acceptable solution for this long-term. Greetings, Sven
So just submit a patch to the tracker... --Guido On Thursday, December 27, 2012, Sven Brauch wrote:
2012/12/27 Nick Coghlan <ncoghlan@gmail.com <javascript:;>>:
It certainly sounds like its worth considering for 3.4. It's a new feature, though, so it unfortunately wouldn't be possible to backport it to any earlier releases.
Yes, that is understandable. It wouldn't be much of a problem tough, my whole project is pretty bleeding-edge anyways, and depending on python >= 3.4 wouldn't hurt. For me it would only be important to have an acceptable solution for this long-term.
Greetings, Sven _______________________________________________ Python-Dev mailing list Python-Dev@python.org <javascript:;> http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org
-- --Guido van Rossum (python.org/~guido)
2012/12/27 Guido van Rossum <guido@python.org>:
So just submit a patch to the tracker...
--Guido
On Thursday, December 27, 2012, Sven Brauch wrote:
2012/12/27 Nick Coghlan <ncoghlan@gmail.com>:
It certainly sounds like its worth considering for 3.4. It's a new feature, though, so it unfortunately wouldn't be possible to backport it to any earlier releases.
Yes, that is understandable. It wouldn't be much of a problem tough, my whole project is pretty bleeding-edge anyways, and depending on python >= 3.4 wouldn't hurt. For me it would only be important to have an acceptable solution for this long-term.
Greetings, Sven _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org
-- --Guido van Rossum (python.org/~guido)
I submitted a patch to the tracker, see http://bugs.python.org/issue16795. The patch only contains the very minimum set of changes which are necessary. There's still a few things you'll need to work around when writing a static language analyzer, but none of them is too much work, so I didn't include them for now in order to keep things compact. Thanks and best regards, Sven
2012/12/27 Sven Brauch <svenbrauch@googlemail.com>:
2012/12/27 Guido van Rossum <guido@python.org>:
So just submit a patch to the tracker...
--Guido
On Thursday, December 27, 2012, Sven Brauch wrote:
2012/12/27 Nick Coghlan <ncoghlan@gmail.com>:
It certainly sounds like its worth considering for 3.4. It's a new feature, though, so it unfortunately wouldn't be possible to backport it to any earlier releases.
Yes, that is understandable. It wouldn't be much of a problem tough, my whole project is pretty bleeding-edge anyways, and depending on python >= 3.4 wouldn't hurt. For me it would only be important to have an acceptable solution for this long-term.
Greetings, Sven _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org
-- --Guido van Rossum (python.org/~guido)
I submitted a patch to the tracker, see http://bugs.python.org/issue16795. The patch only contains the very minimum set of changes which are necessary. There's still a few things you'll need to work around when writing a static language analyzer, but none of them is too much work, so I didn't include them for now in order to keep things compact.
Thanks and best regards, Sven
Ping! Nobody complained since I proposed the patch a week ago, just some people added themselves to the notify list. What do you think? Cheers, Sven
participants (3)
-
Guido van Rossum -
Nick Coghlan -
Sven Brauch