Try to understand Coveralls / Coverage output

Hi folks, this is a good opportunity for me to learn to interpret the coverage/coveralls output. In the past I tried coverage on command line for one of my own projects. But I gave up. Maybe I misunderstand the intention of coverage? You write a "common/backintime.py". Then you have a "common/test/test_backintime.py". And the goal is that the tests in "test_backintime.py" do call every function and method that exists in "backintime.py". Am I right? If that is the case you have 100% and coverage is green. Correct? No look at my last PR and click further to the coveralls output at the end. https://github.com/bit-team/backintime/pull/1305 https://coveralls.io/builds/52681188 Lets check for "common/configfile.py". This one file I know where no "test_configfile.py" exists. So I expect it to have 0%. No tests no coverage. My problem with the coveralls output - The files in "common" folder are not shown on coveralls or I can't find them. - All "test*.py" files are listed on coveralls and have nearly 100% each. Makes no sense to me because I don't test the test files. I don't care if the test files are covered by other tests. ;) And even if I would calculate the coverage of the test file code I would expect nearly 0% because no one have ever written code to test the test files. When using coverage on command line in the past with another project I also had the problem that the "coverage" of third-party packages are shown. I don't care about the coverage of packages I installed form PyPi or the distro repository. I'm confused. ;) Kind Christian

On Thu, 2022-09-22 at 19:32 +0000, c.buhtz@posteo.jp wrote:
Maybe I misunderstand the intention of coverage? You write a "common/backintime.py". Then you have a "common/test/test_backintime.py". And the goal is that the tests in "test_backintime.py" do call every function and method that exists in "backintime.py". Am I right? If that is the case you have 100% and coverage is green. Correct?
Correct (if you mean 100 % coverage for "backintime.py")
No look at my last PR and click further to the coveralls output at the end. https://github.com/bit-team/backintime/pull/1305 https://coveralls.io/builds/52681188 [...] My problem with the coveralls output - The files in "common" folder are not shown on coveralls or I can't find them.
Click on the "List" tab to see all files (also set "Show 50 Entries" to see all files at once). Then click on "configfile.py" which then shows the coverage per line of code: https://coveralls.io/builds/52681188/source?filename=configfile.py
Lets check for "common/configfile.py". This one file I know where no "test_configfile.py" exists. So I expect it to have 0%. No tests no coverage.
It depends: - Correct if and if only every unit test really only calls one basic "unit" (function) and this function does does not call any other function (not very realistic ;-) - If any function called by a any unit test calls your function (in configfile.py) the code coverage is > 0 % (in your branch: 94,59 %). In practice many unit tests have certain "integration test side-effect" or have in case of test_backintime.py even a complete end-to-end use case test character (eg. "take a snapshot). You may wonder: Do you still need a separate "test_configfile.py"? Yes, if either a) the code coverage of your function/file is below 80 + x % (x is subject of a code quality policy) b) no actual/expected assumptions are checked in the unit tests (remember: If you drop ALL assumptions from your unit tests you still have 100 % code coverage but a severe loss of test quality because you only check for syntax compiling/interpretation errors but not for valid results!)
All "test*.py" files are listed on coveralls and have nearly 100% each. Makes no sense to me because I don't test the test files. [...] When using coverage on command line in the past with another project I also had the problem that the "coverage" of third-party packages are shown.
This is why I have opened https://github.com/bit-team/backintime/issues/1282 and implemented the "make coverage" target which includes only the code files (but no unit test files and no libs). And: - Travis/Coveralls is to track the coverage over time (and maybe **centrally** ensure a growing coverage for each commit eg. as PR merge policy) - "make coverage" is to develop unit tests on your local machine HIH
participants (2)
-
c.buhtz@posteo.jp
-
J. A.