Disable Codecov on GitHub pull requests?
Hi,
I don't get the value of code coverage on a CI. I *like* running code coverage sometimes, but I don't like being annoying by "test failed: coverage -0,01%" by a change completely unrelated to code (note: this issue has been fixed be using a threshold of 1%). I'm annoyed by all these Codecov messages. Berker just told me that he even created a Gmail filter to drop all these email notifications...
What do you think of keeping this very useful feature, but not use it on pull requests: only run it on the branches (once changes are merged), to not "pollute" pull requests?
Victor
I'm obviously not very active in CPython development these days, but actively tracking the impact of each PR on coverage has been extremely useful in every other project I've worked on.
It's the best way to automatically ensure PRs are not regressing coverage too much, and doesn't require much manual work. FWIW, on projects I've worked on we have turned off the codecov "comments", and instead just rely on the status checker.
Alex
On Mon, Feb 13, 2017 at 9:49 AM, Victor Stinner victor.stinner@gmail.com wrote:
Hi,
I don't get the value of code coverage on a CI. I *like* running code coverage sometimes, but I don't like being annoying by "test failed: coverage -0,01%" by a change completely unrelated to code (note: this issue has been fixed be using a threshold of 1%). I'm annoyed by all these Codecov messages. Berker just told me that he even created a Gmail filter to drop all these email notifications...
What do you think of keeping this very useful feature, but not use it on pull requests: only run it on the branches (once changes are merged), to not "pollute" pull requests?
Victor
python-committers mailing list python-committers@python.org https://mail.python.org/mailman/listinfo/python-committers Code of Conduct: https://www.python.org/psf/codeofconduct/
-- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero GPG Key fingerprint: D1B3 ADC0 E023 8CA6
On Feb 13, 2017, at 09:51 AM, Alex Gaynor wrote:
actively tracking the impact of each PR on coverage has been extremely useful in every other project I've worked on.
Agreed.
FWIW, on projects I've worked on we have turned off the codecov "comments", and instead just rely on the status checker.
Also agreed. On many projects we do two things, both of which are usually implemented as status checks without the coverage comments. We use coverage to enforce a ratchet so coverage doesn't regress (assuming a stable test suite), and we use diffcov to ensure that any new code being proposed is 100% covered.
Another useful tool that doesn't exist yet, though there were some conversations about it at a previous Pycon is a tool that ensures that a test for the new code has been added. It would do this by running the test suite without the (non-test-suite) change and proving that the test failed, then reapply the fix and prove that the test suite passed. It's unfortunately not uncommon for someone to submit a PR with a test that they *think* tests the new code but actually doesn't, and the only way to check that now is locally and manually.
https://github.com/paulcollinsiii/hasregression
+1 for suppressing the coverage comments.
Cheers, -Barry
On Feb 13, 2017, at 9:49 AM, Victor Stinner victor.stinner@gmail.com wrote:
Hi,
I don't get the value of code coverage on a CI. I *like* running code coverage sometimes, but I don't like being annoying by "test failed: coverage -0,01%" by a change completely unrelated to code (note: this issue has been fixed be using a threshold of 1%). I'm annoyed by all these Codecov messages. Berker just told me that he even created a Gmail filter to drop all these email notifications...
What do you think of keeping this very useful feature, but not use it on pull requests: only run it on the branches (once changes are merged), to not "pollute" pull requests?
I think it’s incredibly useful on PRs, particularly because it can tell you if the PR has actually covered every line that it’s added or not. If you’re only periodically running coverage, then in my experience you generally miss covering a lot of lines accidentally. I’ve also never see the random -0.01% coverage of code in another project, and my guess is that there is some sort of non-determinism in the CPython test suite that would be a good idea to make deterministic anyways.
As with Alex, most projects I’ve been involved in turn off the comments and rely on the status checker.
— Donald Stufft
2017-02-13 16:04 GMT+01:00 Donald Stufft donald@stufft.io:
I’ve also never see the random -0.01% coverage of code in another project, and my guess is that there is some sort of non-determinism in the CPython test suite that would be a good idea to make deterministic anyways.
Can we hide/disable Codecov notifications until this issue is fixed? Example of false alarm: https://github.com/python/cpython/pull/61#issuecomment-279291824 heapq.py coverage -0.77% on a change modifying Doc/library/typing.rst
See also: "Adjust coverage failure threshold" https://github.com/python/core-workflow/issues/21
Victor
On Mon, 13 Feb 2017 at 07:26 Victor Stinner victor.stinner@gmail.com wrote:
2017-02-13 16:04 GMT+01:00 Donald Stufft donald@stufft.io:
I’ve also never see the random -0.01% coverage of code in another project, and my guess is that there is some sort of non-determinism in the CPython test suite that would be a good idea to make deterministic anyways.
Can we hide/disable Codecov notifications until this issue is fixed? Example of false alarm: https://github.com/python/cpython/pull/61#issuecomment-279291824 heapq.py coverage -0.77% on a change modifying Doc/library/typing.rst
See also: "Adjust coverage failure threshold" https://github.com/python/core-workflow/issues/21
Codecov's configuration is controlled by https://github.com/python/cpython/blob/master/.codecov.yml so you can submit a PR to propose changes to how coverage is reported. Docs can be found at https://docs.codecov.io/docs/codecov-yaml .
On Mon, Feb 13, 2017 at 1:11 PM, Brett Cannon brett@python.org wrote:
On Mon, 13 Feb 2017 at 07:26 Victor Stinner victor.stinner@gmail.com wrote:
2017-02-13 16:04 GMT+01:00 Donald Stufft donald@stufft.io:
I’ve also never see the random -0.01% coverage of code in another project, and my guess is that there is some sort of non-determinism in the CPython test suite that would be a good idea to make deterministic anyways.
Can we hide/disable Codecov notifications until this issue is fixed? Example of false alarm: https://github.com/python/cpython/pull/61#issuecomment-279291824 heapq.py coverage -0.77% on a change modifying Doc/library/typing.rst
See also: "Adjust coverage failure threshold" https://github.com/python/core-workflow/issues/21
Codecov's configuration is controlled by https://github.com/python/cpython/blob/master/.codecov.yml so you can submit a PR to propose changes to how coverage is reported. Docs can be found at https://docs.codecov.io/docs/codecov-yaml .
Berker has https://github.com/python/cpython/pull/71 up right now. I think it looks ok so far from what I understand.
On Mon, Feb 13, 2017 at 6:25 PM, Victor Stinner victor.stinner@gmail.com wrote:
2017-02-13 16:04 GMT+01:00 Donald Stufft donald@stufft.io:
I’ve also never see the random -0.01% coverage of code in another project, and my guess is that there is some sort of non-determinism in the CPython test suite that would be a good idea to make deterministic anyways.
Can we hide/disable Codecov notifications until this issue is fixed? Example of false alarm: https://github.com/python/cpython/pull/61#issuecomment-279291824 heapq.py coverage -0.77% on a change modifying Doc/library/typing.rst
See also: "Adjust coverage failure threshold" https://github.com/python/core-workflow/issues/21
I've opened to tweak our Codecov configuration: https://github.com/python/cpython/pull/71
--Berker
participants (7)
-
Alex Gaynor -
Barry Warsaw -
Berker Peksağ -
Brett Cannon -
Brian Curtin -
Donald Stufft -
Victor Stinner