On Feb 13, 2017, at 9:49 AM, Victor Stinner <victor.stinner@gmail.com> wrote:

Hi,

I don't get the value of code coverage on a CI. I *like* running code
coverage sometimes, but I don't like being annoying by "test failed:
coverage -0,01%" by a change completely unrelated to code (note: this
issue has been fixed be using a threshold of 1%). I'm annoyed by all
these Codecov messages. Berker just told me that he even created a
Gmail filter to drop all these email notifications...

What do you think of keeping this very useful feature, but not use it
on pull requests: only run it on the branches (once changes are
merged), to not "pollute" pull requests?

I think it’s incredibly useful on PRs, particularly because it can tell you if the PR has actually covered every line that it’s added or not. If you’re only periodically running coverage, then in my experience you generally miss covering a lot of lines accidentally. I’ve also never see the random -0.01% coverage of code in another project, and my guess is that there is some sort of non-determinism in the CPython test suite that would be a good idea to make deterministic anyways.