On Mon, 29 Jul 2013 13:58:55 -0400, Brett Cannon <brett@python.org> wrote:

On Mon, Jul 29, 2013 at 1:15 PM, Christian Heimes <christian@python.org>wrote:

...

Hi,

I have done some experiments with GCC's gcov and lcov to get the C code coverage of our unit test suite. You may find today's report at

http://tiran.bitbucket.org/python-lcov/

Thanks!

I took a quick poke around and it seems some things are legitimately not being executed, while others are error conditions that we wouldn't expect to occur (e.g. memory exhaustion). If we ever decide to get serious about code coverage (both C and Python code) we may need to have a discussion as a group about our feelings related to pragmas dictating when code should be left out of coverage reports.

I suspect Victor will eventually have something for us for exercising the memory exhaustion code :) --David

On Mon, Jul 29, 2013 at 3:31 PM, Christian Heimes <christian@python.org>wrote:

Am 29.07.2013 19:58, schrieb Brett Cannon:

...

I took a quick poke around and it seems some things are legitimately not being executed, while others are error conditions that we wouldn't expect to occur (e.g. memory exhaustion). If we ever decide to get serious about code coverage (both C and Python code) we may need to have a discussion as a group about our feelings related to pragmas dictating when code should be left out of coverage reports.

Yeah, object allocation and creation checks as well as verification of operating system API call are reason for lots branch and line misses. I'm not sure what we can do about that. I don't want to plaster the code with LCOV_EXCL_LINE comments.

As first action we should look into function coverage. We may not be able to execute every branch but at least we should be able to call and verify each function.

If there's a way to report just function coverage then I think that's a great place to start.

On Mon, 29 Jul 2013 21:31:02 +0200 Christian Heimes <christian@python.org> wrote:

Am 29.07.2013 19:58, schrieb Brett Cannon:

...

I took a quick poke around and it seems some things are legitimately not being executed, while others are error conditions that we wouldn't expect to occur (e.g. memory exhaustion). If we ever decide to get serious about code coverage (both C and Python code) we may need to have a discussion as a group about our feelings related to pragmas dictating when code should be left out of coverage reports.

Yeah, object allocation and creation checks as well as verification of operating system API call are reason for lots branch and line misses. I'm not sure what we can do about that. I don't want to plaster the code with LCOV_EXCL_LINE comments.

Ideally, we should run coverage runs on different systems (Unices, Windows...) and merge the results together, so that we know which paths are really uncovered. Regards Antoine.

On Tue, Jul 30, 2013 at 8:28 PM, Christian Heimes <christian@python.org>wrote:

Am 29.07.2013 21:48, schrieb Antoine Pitrou:

...

Ideally, we should run coverage runs on different systems (Unices, Windows...) and merge the results together, so that we know which paths are really uncovered.

I don't that is easily possible. The coverage report depends on GCC and its gcov extension -- so much for Windows. I also don't know if gcov supports cross-profiling on varying platforms and operating systems.

By the way gcov understands preprocessor output. It doesn't report lines as uncovered when the lines or functions are #ifdef-ed out.

In case are curious to try and make this work using clang, I found the flags to use at http://clang-developers.42468.n3.nabble.com/Code-coverage-on-clang-td4033066... and how to make it work with lcov. I also discovered clang itself uses https://github.com/ddunbar/zcov instead of lcov. This is all untested, but I didn't want to forget the links and in case someone has clang installed with compiler-rt and wants to see how it works.