On 2/22/2021 6:58 AM, Victor Stinner wrote:
On Mon, Feb 22, 2021 at 12:51 PM Ivan Pozdeev via Python-Dev firstname.lastname@example.org wrote:
IIRC I suggested earlier that buildsbots should be integrated into the PR workflow in order to make it the contributor's rather than a core dev's burden to fix any breakages that result from their changes.
The problems is that many 'breakages' do NOT result from the PR changes. For IDLE patches, EVERY CI failure other than in test_idle is a bogus failure. My productivity is already degraded by the current blind automated rules.
My point here is that machines literally have no judgment, and that automation is not the panacea that people think. Writing good rules is often hard to impossible, and blind application of inadequate rules leads to bad results.
Some buildbot worker take 2 to 3 hours per build. Also, it would not scale. Buildbots are not fast enough to handle the high number of PR and PR updates.
When there is clear relationship between a buildbot failure and a merged PR, a comment is added automatically showing the failed test and explanation how to investigate the issue.
This is false and an example of the above comment. The notice is sent whenever any buildbot test fails after a merge, regardless of whether there is any relationship or not. There often (usually?) is not.
It's there for 2 years thanks to Pablo, and so far, I rarely saw developers paying attention to these failures. They just ignore it.
Every one of those big, bold, ugly notices I have received has been a false positive. I believe they are usually due to one of the flaky tests (asyncio or otherwise) that should have been disabled but have not been. Of course people ignore routinely false positives.
If there is an expert for the module whose test file failed, that person should be the target of the notice. If a PR or merge breaks test_idle, I would like to know.
Actually, because I am conscientious, and because there was one instance years ago where test_idle passed CI and failed on one buildbot, and because the notice lists which test file failed (in among the noise), I do check that line before deleting the notice.
I'm not trying to blame anyone. Contributing to Python requires a lot of free time. I'm only trying to explain in length what does the "maintenance burden" mean in practice, since some people are pretending that supporting a platform is free.
Blind automated blind rules are also not free.