Quoting Andrew Bennetts <andrew-twisted@puzzling.org>:
Thomas Hervé wrote: [...]
[1] I'll be even happier when we have *machines* monitoring the test suite, but that's another email.
Currently, our test suite is a little bit too unreliable to do this (aka 'intermittent failures'). But maybe in a near future...
We've had intermittent failures for *years*. I doubt that'll get solved in a near future without some sort of significant change to how we do things.
Here's a thought experiment: one possibility would be completely automate the running of the test suite, as jml is suggesting, e.g. by using a commit hook (or a ?commit bot? like PQM) that will run the test suite before allowing a commit to trunk. Then intermittent failures become much more intolerable, because they will regularly frustrate developers rather than being something you can usually ignore. Thus people will be much more motivated to find solutions to our intermittent tests (whether by fixing the offending tests, or disabling them, or something else).
I'm not sure it'd be worth the effort, but it's interesting to think about...
I think it definitely worth the effort. As I see it, maybe we could split the effort in 2 parts: * having one commit bot for our best supported platform. I think debian-py2.4-select has been free of intermittent failure for a bit now. Maybe we could trust it enough to disallow commits that breaks it. The other advantage for this is that we could formalize the merge process a bit more: for example, with a web page where you specify the branch name, authors, reviewers, tickets fixed, etc. It looks like what PQM is doing. * having better notifications for trunk commits that generate failed runs. Today, this work is basically done by JP looking at the buildbot and remembering the failures and tickets linked to them, and possibly creating a new ticket if it's not an already identified problem. It's not only cumbersome but probably let some errors pass through. We could look at this after the release :). -- Thomas