[Python-Dev] How to watch buildbots?

[Victor Stinner]

2018-05-30 14:30 GMT+02:00 Ivan Pozdeev via Python-Dev <python-dev at python.org>:
> What's the big idea of separate buildbots anyway? I thought the purpose of
> CI is to test everything _before_
> it breaks the main codebase. Then it's the job of the contributor rather
> than maintainer to fix any breakages.

I will answer more generally.

Technically, buildbots support to send emails to author of changes
which introduced a regression.

But a build may test a single change or dozens of new changes.

Moreover, our test suite is not perfect: they are at least 5 known
tests which fail randomly. Even if we fix these unstable tests, it's
also "common" that buildbots fail for "external" reasons:

* network failure: fail to clone the GitHub repository

* functional test using an external service and the service is down. I
started to list external services used by "unit" tests:
http://vstinner.readthedocs.io/cpython.html#services-used-by-unit-tests

* vacuum cleaner:
https://mail.python.org/pipermail/python-buildbots/2017-June/000122.html

* many other random reasons...

Since two years, I'm trying to fix all tests failing randomly, but as
I just explained, it's really hard to get a failure rate of 0%.

I'm not sure that we can "require" authors of pull requests to
understand buildbot failures...

So I prefer to keep the status quo: filter buildbot failures manually.

Victor