Thanks Victor and the rest of the Night’s Watch for all the work in our buildbots!
El jue., 3 de sep. de 2020 a la(s) 12:08, Victor Stinner (vstinner@python.org) escribió:
Hi,
tl; dr Buildbots were unstable for 3 weeks but the issue is mostly resolved.
Since last January, the disk of the buildbot server was full every 2 weeks and The Night’s Watch had to fix it in the darkness for you (usually, remove JUnit files and restart the server). The old machine only has 8 GB for the whole system and all data, whereas buildbot workers produce large JUnit (XML) files (around 5 MB per file).
Three weeks ago, Ernest W. Durbin III provided us a new machine with a larger disk (60 GB) and installed PostgreSQL database (whereas SQLite was used previously). He automated the installation of the machine, but also (great new feature!) automated reloading the Buildbot server when a new configuration is pushed in the Git repository. The configuration is public and maintained at: https://github.com/python/buildmaster-config/
The migration was really smooth, except that last week, we noticed that workers started to be disconnected every minute, and then filled their temporary directory with temporary compiler files leaked by interrupted builds. Buildbot owners have to update their client configuration and remove manually temporary files: https://mail.python.org/archives/list/python-buildbots@python.org/thread/SZR...
Most buildbot worker configurations have been updated and the issue is mostly resolved.
There is another minor issue, HTTPS connections also closed after 1 minute and so the web page is refreshed automatically every minute. The load balancer configuration should be adjusted: https://bugs.python.org/issue41701
Victor
Night gathers, and now my watch begins. It shall not end until my death. _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/DYUX5EED... Code of Conduct: http://python.org/psf/codeofconduct/