Thinking about the discussion I (inadvertently) started about what sorts of issues should cause tools like pylint and pyflakes to emit messages, got me thinking about how I try to use these tools. I don't know what the current suite of all available static checking tools is, but let me just assume for the moment that we have three tools: pylint, pyflakes, and (the probably now defunct) pychecker. Each one has different goals, which naturally affects the messages it can possibly emit. For example, because pychecker imports the modules mentioned in the code it analyzes, it is able to build symbol tables for extension modules. OTOH, it could be slower, the user might suffer if importing modules had side effects, and to get the most out of it, you were tied to the platforms for which you had compiled extension modules. I don't think pylint actually imports the code it analyzes, so it can be more platform-independent, but not as comprehensive as pychecker when trying to emit warnings about cross-module issues. It can, however, identify other problematic constructs. As a user of these tools, recognizing that none will ever be "complete", I try to use multiple tools precisely so I can see the union of all messages. In fact, some while ago (at least a couple years, probably longer), I wrote a shell script wrapper which invokes both pychecker and pylint, presenting me with the sorted (and very lightly massaged) output. If pyflakes is so conservative in what it will emit that it tries to avoid all possible false positives, it's less useful to me, precisely because the odds are low that it will report problems the other available tools won't catch. That is: pylint ∪ pychecker ≈ pylint ∪ pychecker ∪ pyflakes making the incremental benefit to me of using pyflakes small. This is not to say that Phil doesn't have excellent reasons for his approach, just that there is a non-trivial trade-off in that approach. Skip
Hi Skip, This kind of argument was on my mind a little when I made prospector (https://github.com/landscapeio/prospector) although probably from a different angle. There are many tools out there and using them all means you can get the maximal coverage of all concepts, with some overlap. Each tool has a spectrum of certainty and usefulness, and what you really want is the biggest problems in a specific code base. "Biggest problem" as a concept varies from codebase to codebase. Of course the ideal tool figures out exactly what you want and what level of scrutiny you want on your code, and delivers the most important problems first. Such a tool is impossible! I think you can get there though, if you merge each tool (and therefore, each opinion of what is important) enough. The main concern is how to filter the wheat from the chaff. So far I have simply made opinionated defaults and there is a way to make your own configuration; that level of effort isn't that different to simply configuring each tool yourself! I'd love some feedback on this concept in general. I think each tool and each error and each codebase and each user is a unique point and it's difficult to create something that caters to all tastes. However, having several starting positions tailored to individual use cases could be helpful. Cheers, Carl On 8 December 2014 at 22:10, Skip Montanaro <skip.montanaro@gmail.com> wrote:
Thinking about the discussion I (inadvertently) started about what sorts of issues should cause tools like pylint and pyflakes to emit messages, got me thinking about how I try to use these tools. I don't know what the current suite of all available static checking tools is, but let me just assume for the moment that we have three tools: pylint, pyflakes, and (the probably now defunct) pychecker. Each one has different goals, which naturally affects the messages it can possibly emit. For example, because pychecker imports the modules mentioned in the code it analyzes, it is able to build symbol tables for extension modules. OTOH, it could be slower, the user might suffer if importing modules had side effects, and to get the most out of it, you were tied to the platforms for which you had compiled extension modules. I don't think pylint actually imports the code it analyzes, so it can be more platform-independent, but not as comprehensive as pychecker when trying to emit warnings about cross-module issues. It can, however, identify other problematic constructs.
As a user of these tools, recognizing that none will ever be "complete", I try to use multiple tools precisely so I can see the union of all messages. In fact, some while ago (at least a couple years, probably longer), I wrote a shell script wrapper which invokes both pychecker and pylint, presenting me with the sorted (and very lightly massaged) output. If pyflakes is so conservative in what it will emit that it tries to avoid all possible false positives, it's less useful to me, precisely because the odds are low that it will report problems the other available tools won't catch. That is:
pylint ∪ pychecker ≈ pylint ∪ pychecker ∪ pyflakes
making the incremental benefit to me of using pyflakes small.
This is not to say that Phil doesn't have excellent reasons for his approach, just that there is a non-trivial trade-off in that approach.
Skip
_______________________________________________ code-quality mailing list code-quality@python.org https://mail.python.org/mailman/listinfo/code-quality
On Mon, Dec 8, 2014 at 4:10 PM, Skip Montanaro <skip.montanaro@gmail.com> wrote:
This is not to say that Phil doesn't have excellent reasons for his approach, just that there is a non-trivial trade-off in that approach.
It's a feature, not a bug. You can run pyflakes against some ugly code that an intern wrote 3 years ago, and every single thing it tells you is a bug in the program (or a bug in pyflakes). It won't complain about how the intern didn't understand that classes should be named in CamelCase, or gripe that every iteration variable is named "i", even though the code works. Without any up-front effort or configuration, pyflakes will point you immediately at the areas of code that have problems instead of sending you off on a 2-week search-and-replace mission to rename all the variables and twiddle whitespace which is more likely to introduce bugs than to fix them. The premise of pyflakes is that you are smarter than your linting tool, you write tests to check the correctness of your program, and you audit for readability in code review. You can also run it against really good code without any configuration effort, and still everything you get is a real problem in the code, and pyflakes will find things that a unit test will not. In fact, when I wrote pyflakes, pylint and pychecker already existed. I was working at Divmod, which was staffed by many of the core contributors of Twisted. You can look at the Twisted code to get an idea of the climate: unit test coverage is nearly 100%, there is a brutal review process, and all the developers have years of experience and not a single intern in sight. The code quality is very high. However, we had a few problems where sometimes we'd add an import to a file, and suddenly the module couldn't be imported because of a cycle in the import dependencies. Frequently, the cycle was due to an import which served no purpose, maybe left over from a function that had moved elsewhere. So, the first and easiest step to reducing the cycles would be to remove all these unnecessary imports. I tried pychecker and pylint, and they each had problems: 1) they were slow. 2) they spewed *thousands* of "problems". Remember, this is a code base that is super well tested. There weren't thousands of problems. I suppose I could have taken the time to figure out how to configure these tools to not complain about how Divmod's naming conventions didn't match the tool's defaults, and disable the dozens of other "style suggestions" and warnings about things that "might" be bad but which, because of the tests and code review, I knew were not. But given that the existing tools were already slow, I figured I could do better, and pyflakes was born. After I wrote pyflakes and we ran it against the code, we also found some code paths which were useless. They didn't show up in unit test code coverage reports because they were executed, but maybe they'd calculate a variable that never got used. We were then able to remove these bits of useless code. Between Divmod and Twisted I had a large corpus of well-tested, rigorously reviewed, high-quality code, so I reasoned that if pyflakes found a false-positive in this code, it was a bug in pyflakes. So, I'd argue that the incremental benefit of using pylint is small if you have tests. I should also point out that a lot of people seems to agree with this philosophy, but still want to enforce conformance to some kind of naming and whitespace convention, like PEP8. There's a program that does exactly that, called (unsurprisingly) pep8. There's also flake8, which combines pyflakes and pep8. Based on PyPI download stats, It's a fair guess that the most popular use of pyflakes is as a dependency of flake8.
participants (3)
-
Carl Crowder -
Phil Frost -
Skip Montanaro