This, and an earlier letter about the burned of a release manager, confirmes my suspicion that a large share of "maintenance burden" comes from unresolved process issues. E.g. looks like Victor had been unconvinced that unreliable tests are a serious issue and cost the team so much (including himself since they cause ppl to ignore his buildbot issue reports right back) and had blocked the effort to disable them or whatnot (this is just my guess but Terry's message suggests that he was against resolving it). I hope this convinces him and will let the team clear this issue at last! Likewise, half of the bullet points in https://mail.python.org/archives/list/python-dev@python.org/message/LJ3E2UQB... comes from either ppl trying to bypass the process, or the release manager doing others' jobs that should've already been done had the process been followed. On 22.02.2021 19:48, Terry Reedy wrote:
On 2/22/2021 6:20 AM, Victor Stinner wrote:
To have an idea of the existing maintenance burden, look at emails sent to: https://mail.python.org/archives/list/buildbot-status@python.org/
Every single email is basically a problem. There are around 110 emails over the last 30 years:
30 days, not years. The 20 visible messages are all have title f"Buildbot worker {worker} missing". All the messages on a day are sent at the same time. Replace with a daily "Builbot workers currently missing"? The individual messages seem useless except possibly if sent to individual buildbot owners. ...
Multiple buildbots are "unstable": tests are failing randomly. Again, each failure means a new email. For example, test_asyncio likes to fail once every 10 runs (coarse average, I didn't check exactly).
I strongly feel that the individual repeatedly failing tests (only 3 or 4 I think) should be disabled and the asyncio people notified. As I remember, you once insisted on keeping routinely failing tests.
By the way, these random failures are not only affecting buildbots, but also CIs run on pull requests. It's *common* that these failures prevent to merge a pull request and require manual actions to be able to merge the PR (usually, re-run all CIs).
For backports, it means closing the backport and reopening by re-adding a backport label to the master PR. This is why I really really want the repeatedly failing tests disabled.
-- Regards, Ivan