Am 26.07.2013 00:32, schrieb Terry Reedy:

I found the answer here https://docs.google.com/file/d/0B5wQCOK_TiRiMWVqQ0xPaDEzbkU/edit Coverity Integrity Level 1 is 1 (defect/1000 lines) Level 2 is .1 (we have passed that) Level 3 is .01 + no major defects + <20% (all all defects?) false positives as that is their normal rate.#

A higher false positive rates requires auditing by Coverity. They claim "A higher false positive rate indicates misconfiguration, usage of unusual idioms, or incorrect diagnosis of a large number of defects." They else add "or a flaw in our analysis."

# Since false positives should stay constant as true positives are reduced toward 0, false / all should tend toward 1 (100%) if I understand the ratio correctly.

About 40% of the dismissed cases are cause by a handful of issues. I have documented these issues as "known limitations" http://docs.python.org/devguide/coverity.html#known-limitations . For example about 35 false positives are related to PyLong_FromLong() and our small integer optimization. A correct modeling file would eliminate the false positive defects. My attempts don't work as hoped and I don't have access to all professional coverity tools to debug my trials. Nearly 20 false positives are caused by Py_BuildValue("N"). I'm still astonished that Coverity understands Python's reference counting most of the time. :) Did I mention that we have almost reached Level 3? All major defects have been dealt with (one of them locally on the test machine until Larry pushes his patch soonish), 4 of 7 minor issues must be closed and our dismissed rate is just little over 20% (222 out of 1054 = 21%). Christian

On Thu, Jul 25, 2013 at 6:56 PM, Christian Heimes wrote:

Am 26.07.2013 00:32, schrieb Terry Reedy:

...

I found the answer here https://docs.google.com/file/d/0B5wQCOK_TiRiMWVqQ0xPaDEzbkU/edit Coverity Integrity Level 1 is 1 (defect/1000 lines) Level 2 is .1 (we have passed that) Level 3 is .01 + no major defects + <20% (all all defects?) false positives as that is their normal rate.#

A higher false positive rates requires auditing by Coverity. They claim "A higher false positive rate indicates misconfiguration, usage of unusual idioms, or incorrect diagnosis of a large number of defects." They else add "or a flaw in our analysis."

# Since false positives should stay constant as true positives are reduced toward 0, false / all should tend toward 1 (100%) if I understand the ratio correctly.

About 40% of the dismissed cases are cause by a handful of issues. I have documented these issues as "known limitations" http://docs.python.org/devguide/coverity.html#known-limitations .

For example about 35 false positives are related to PyLong_FromLong() and our small integer optimization. A correct modeling file would eliminate the false positive defects. My attempts don't work as hoped and I don't have access to all professional coverity tools to debug my trials.

Have you tried asking for help from Coverity? They have been rather nice so far and they may be willing to just give us free help in getting the modeling file set up properly. -Brett

Nearly 20 false positives are caused by Py_BuildValue("N"). I'm still astonished that Coverity understands Python's reference counting most of the time. :)

Did I mention that we have almost reached Level 3? All major defects have been dealt with (one of them locally on the test machine until Larry pushes his patch soonish), 4 of 7 minor issues must be closed and our dismissed rate is just little over 20% (222 out of 1054 = 21%).

On Thu, 25 Jul 2013 18:00:55 -0400 Terry Reedy wrote:

On 7/25/2013 2:48 PM, Christian Heimes wrote:

...

Hello,

this is an update on my work and the current status of Coverity Scan.

Great work.

...

Maybe you have noticed a checkins made be me that end with the line "CID #". These are checkins that fix an issue that was discovered by the static code analyzer Coverity. Coverity is a commercial product but it's a free service for some Open Source projects. Python has been analyzed by Coverity since about 2007. Guido, Neal, Brett, Stefan and some other developers have used Coverity before I took over. I fixed a couple of issues before 3.3 reached the RC phase and more bugs in the last couple of months.

The benefit for us is not just improving Python having external verification of its excellence in relation both to other open-source projects and commercial software.

"Excellence"? The term is too weak, I would say "perfection" at least, but perhaps we should go as far as "divinity". Regards Antoine.

Am 26.07.2013 00:00, schrieb Terry Reedy:

...

http://www.coverity.com/company/press-releases/read/coverity-introduces-mont...

The intention is to promote the best of open source to industry.

I think it's also a marketing tool. They like to sell their product. I don't have a problem with that. After all Coverity provides a useful service for free that supplements our own debugging tools.

...

Lines of Code: 396,179

C only? or does Python code now count as 'source code'?

It's just C code and headers. Coverity doesn't analyze Python code. According to cloc Python has 296707 + 78126 == 374833 lines of code in C and header files. I'm not sure why Coverity detects more.

...

Defect Density: 0.05

= defects per thousand lines = 20/400

Anything under 1 is good. The release above reports Samba now at .6. http://www.pcworld.com/article/2038244/linux-code-is-the-benchmark-of-qualit...

reports Linux 3.8 as having the same for 7.6 million lines.

These are amazing numbers. Python is much smaller.

...

Total defects: 1,054 Outstanding: 21 (Coverity Connect shows less) Dismissed: 222

This implies that they accept our designation of some things as False Positives or Intentional. Does Coverity do any review of such designations, so a project cannot cheat?

What's the point of cheating? :) I could dismiss any remaining defect as intentionally or false positive but that would only harm ourselves. As you already pointed out Coverity reserves the right to inspect dismissed bugs for their highest ranking. I'm in the process of looking through all dismissed defects. Some of them are relics of deleted files and removed code. Some other may go away with proper modeling. Christian

Am 26.07.2013 14:56, schrieb Eli Bendersky:

Just a quick question - is there a chance to convince Coverity to detect Python refcounting leaks in C API code :-) ? This could be useful not only for Python but for extensions too. As it stands now, Coverity's leak detection is Python must be pretty weak because almost everything is done via PyObject refcounts.

Coverity is able to detect some cases of refcount leaks. I don't know if the software is able to keep track of all reference counts. But it understands missing Py_DECREF() in error branches. For example: PyObject *n = PyLong_FromLong(0); PyObject *u = PyUnicode_FromString("example"); if (u == NULL) { return NULL; /* Coverity detects that 'n' leaks memory */ } Christian

On Fri, Jul 26, 2013 at 7:29 AM, Christian Heimes wrote:

Am 26.07.2013 14:56, schrieb Eli Bendersky:

...

Just a quick question - is there a chance to convince Coverity to detect Python refcounting leaks in C API code :-) ? This could be useful not only for Python but for extensions too. As it stands now, Coverity's leak detection is Python must be pretty weak because almost everything is done via PyObject refcounts.

Coverity is able to detect some cases of refcount leaks. I don't know if the software is able to keep track of all reference counts. But it understands missing Py_DECREF() in error branches.

For example:

PyObject *n = PyLong_FromLong(0); PyObject *u = PyUnicode_FromString("example");

if (u == NULL) { return NULL; /* Coverity detects that 'n' leaks memory */ }

Interesting. I was thinking of something more general though. Especially if we can mark function arguments and return values as stealing references / creating new ones / etc, many many common refcount bugs can be detected with static analysis. This is definitely research-y, probably too much for our current stage of relationship with Coverity :) Eli