On Wed, Apr 17, 2013 at 8:49 AM, Ian Cordasco <graffatcolmingov@gmail.com> wrote:

On Mon, Apr 15, 2013 at 2:59 PM, Vladimir Keleshev <vladimir@keleshev.com> wrote:

...

Hi Ian,

I'm actually on the mailing list as well (since maybe 2 weeks ago). You might remember my email about proposing a common format for error messages. I think it's a great idea to coordinate development of our static-analysis tools. Adding pep257 to flake8 might be a good idea. Shall we call it fla257ke8? :o)

But I'm not sure I understand the purpose of flake8. Wouldn't having a common error format between all the tools defeat the purpose of flake8? Then a dummy runner would suffice. Imagine a world where all static analysis tools have a common output format and same command-line interface and error conventions.

Cheers, Vladimir

Hey Vladimir,

I'm awful with names, so I apologize for not putting your post there together with the fact you work on PEP257.

flake8 (as soon as we update to pyflakes 0.7 which was just announced) will have a consistent error format. (Prior to 2.0, our vendored copy of pyflakes provided column reporting which was just introduced in pyflakes 0.7 and was missing in every official version prior to it.) As for the purpose of flake8, it was created to run pep8 and pyflakes and optionally check the McCabe complexity score. It is already extensible to check naming conventions via pep8, so extending it (most likely through an add-on) to check doc-strings would be a logical step.

I'm not readily familiar with pep257's error reporting, but in skimming over the code I noticed the Error class. I'm sure if we had to, we could monkey patch your tool (as we already monkey patch pyflakes), to standardize the output for flake8 users. Due to the fact we can do that (and already do), I'm not sure attempting to make a standard way to report errors, warnings, etc. is an immediate priority. It's certainly something to consider, but we will encounter the issue where tools are old enough to have other tools written to parse their output. They could add options for different styles of reporting, but if they're anything like pep8, they already have those options which makes this a non-issue.

Anyway, I look forward to working with you on integrating pep257 into flake8 and I hope to hear more from you.

Cheers, Ian

And after re-reading a couple Flake8 issues, I re-discovered this project: https://github.com/fedora-static-analysis/firehose It was suggested to us by the author itself and you'll probably find it very interesting Vladimir. :) I'm looking into it myself

On Wed, 2013-04-17 at 15:31 +0200, Vladimir Keleshev wrote:

Hi Ian,

I'm ready to assist you with integrating pep257 into flake8. Do you plan to vendorize it or make it a dependency? `Error` is the right class to look for. `__str__` formats the error. The default format looks like:

path/file.py:14:4: PEP257 Class docstrings should have 1 blank line around them.

which looks pretty much the same as flake8's:

coolproject/mod.py:97:1: F401 'shutil' imported but unused

except for dot in the end and no error/warning code. `check_source` will return a list of all errors in a string.

* * *

But still, as I was saying:

...

Wouldn't having a common error format between all the tools defeat the purpose of flake8?

About Firehose: I find it over-engineered. It is radically different from any current error formats. I think that only something simple can succeed when it comes to standards that cannot be enforced.

Hi! I'm the creator of Firehose, so I clearly have a bias here :) I guess the reason for the perceived complexity is that Firehose does a lot of things: * as well as capturing just an issue at a location in a source file, it can also capture a trace of execution to reach the issue. This data is emitter by tools like clang-analyzer and my cpychecker tool, describing a path through the source code that leads to a given problem. (I suspect that the current tools in flake8 don't do that level of data output, so perhaps this complexity is redundant for your use-case) * as well as capturing "issues" in the software under test (SUT), Firehose can catch "failures" meaning a problem with the testing software (e.g. a traceback, potentially associated with the line of code in the SUT that broke the checker): if you have any of these, you don't have full coverage of the software-under-test. I've found this *very* useful when doing mass-runs of my checker on large collections of code: you can record the failures and come back to them later * there's also the concept of "info" about the SUT: something that isn't an "issue" per se, just data. I use this for capturing McCabe complexity of functions in C extension modules in cpychecker. * potentially you may want to add CWE ID for errors, which in Firehose is an optional field of a firehose.model.Issue. cwe.mitre.org is a sister project to CVE, attempting to classify the underlying kinds of coding mistake that lead to security issues (e.g. "CWE-89" which means "SQL Injection vulnerability", as opposed to CVE, which catalogs specific instances of where a CWE occurred in a specific piece of code). ...and so on, this mail is probably too long already. Also most of the firehose data model is optional. But firehose isn't yet API-frozen (neither are the XML or JSON serializations), so if you have ideas for improving things, let me know. In any case, if you have a well-defined format, I'm sure we can write a firehose.parsers module for slurping it into my data model, and a method for getting it back out again. Cheers; hope this is constructive Dave [...snip...]