Hi Buildbot owners,
You need to manually update your buildbot owner clients configuration:
* Change the buildbot server host to: buildbot-api.python.org (it was
buildbot.python.org previously)
* Change the keepalive to: 60 (seconds, it was usually 600 previously)
You should also clean up manually the temporary directory (/tmp) to
remove old "ccXXX" and "tmpXXX" files:
rm -f /tmp/{cc,tmp}*
Stop the client, remove these files, and then start again the client.
For example, on some Red Hat workers, /tmp contained up to 20 GB of
these "temporary" files! These files are leaked by buildbot builds
interrupted by closed TCP connections.
Note: we might pass the "-pipe" option to GCC to avoid leaking
temporary "ccXXX" files when GCC is killed by an interrupted job. But
I'm not sure how to do that.
--
So, what happened?
https://bugs.python.org/issue41642
One week ago, Ernest and Pablo migrated the buildbot server to a new
machine with a larger disk. The old machine only had 8 GB disk and its
disk was frequently full because of the large (JUnit) XML files. The
new machine is now behind a load balancer which is great to have clean
x.509 certificates on HTTPS (https://buildbot.python.org/).
The TCP port 9020 used by buildbot clients also goes through the load
balancer for consistency with the whole PSF infrastructure. The
problem is that the load balancer seems to like closing idle TCP
connections open for longer than 60 seconds. Ernest extended the
timeout to 24 hours, but it seems like the load balancers still closes
TCP connections of buildbot clients (workers).
Ernest exposed the TCP port 9020 directly to the Internet under a new
host name: buildbot-api.python.org. This host is not behind the load
balancer, and so is not affected by the closed TCP connections issue.
Please update your buildbot client configuration to use the new
"buildbot-api.python.org" host. I also suggest to use a keepalive of
60 seconds, it should prevent further TCP connection issue in the
future (if one day, the host goes behind a load balancer again).
Victor
--
Night gathers, and now my watch begins. It shall not end until my death.
