Mailman 3 February 2020 - Python-Buildbots

Buildbot master hung?
by David Edelsohn 02 Mar '20

02 Mar '20

I see a lot of builders not updated since yesterday and showing builds in progress while the builders themselves are idle. - David

3 3

[buildbot@python.org: Buildbot worker billenstein-macos missing]
by Matt Billenstein 27 Feb '20

27 Feb '20

I've received a few of these emails, but every time I check the worker, it appears to be operating normally: mattb@mattb-mbp2:~ $ tail /Users/buildbot/buildarea/twistd.log 2020-02-27 09:25:26-0800 [Broker,client] Master replied to keepalive, everything's fine 2020-02-27 09:36:27-0800 [-] sending app-level keepalive 2020-02-27 09:36:27-0800 [Broker,client] Master replied to keepalive, everything's fine 2020-02-27 09:47:09-0800 [-] sending app-level keepalive 2020-02-27 09:47:09-0800 [Broker,client] Master replied to keepalive, everything's fine 2020-02-27 09:57:55-0800 [-] sending app-level keepalive 2020-02-27 09:57:55-0800 [Broker,client] Master replied to keepalive, everything's fine 2020-02-27 10:08:33-0800 [-] sending app-level keepalive 2020-02-27 10:08:33-0800 [Broker,client] Master replied to keepalive, everything's fine 2020-02-27 12:17:03-0800 [-] sending app-level keepalive mattb@mattb-mbp2:~ $ date Thu Feb 27 13:23:56 PST 2020 mattb@mattb-mbp2:~ $ date --utc Thu Feb 27 21:23:57 UTC 2020 Did something change on the master or somewhere else that's causing these emails? m -- Matt Billenstein matt(a)vazor.com http://www.vazor.com/

1 1

aarch64 workers ready for joining the CPython buildbot fleet
by Charalampos Stratakis 20 Feb '20

20 Feb '20

Hello everyone, Through the Works on Arm initiative [0], packet.com and arm.com have provided a bare metal aarch64 server for usage with the Python CI system. Specs: 32 Physical Cores @ 3.3 GHz (1 x Ampere eMag), 128 GB of DDR4 ECC RAM, 480 GB SSD and Dual Port Mellanox NIC (2 × 10Gbps w/ LACP) The server is running CentOS 8, and I have created four VM's each one with 8 vCores, 32GB of ram and 50GB of hdd space. The VM's are running Fedora rawhide, Fedora stable, RHEL 7 and RHEL 8 respectively. I've already set up the workers and sent a PR for deploying them to the buildbot master: https://github.com/python/buildmaster-config/pull/169/files All the new builders are added to the unstable list for the time being, as there haven't been any arm 64 buildbots so far, so I would like to test how things go first. [0] https://www.worksonarm.com/ -- Regards, Charalampos Stratakis Software Engineer Python Maintenance Team, Red Hat

2 2

s390x Rawhide won't connect
by David Edelsohn 12 Feb '20

12 Feb '20

I have been trying to keep s390x rawhide up to date. An update yesterday caused the system to complain about inability to resolve github.com although it retained the connection to buildbot master. The system then wedged. We could not boot with the current kernel. Reverting to the previous kernel boots, but now I receive the following error: Following twistd.log until startup finished.. Removing stale pidfile /home/dje/cpython-buildarea/twistd.pid The worker took more than 10 seconds to start and/or connect to the buildmaster, so we were unable to confirm that it started and connected correctly. Please 'tail twistd.log' and look for a line that says 'message from master: attached' to verify correct startup. If you see a bunch of messages like 'will retry in 6 seconds', your worker might not have the correct hostname or portnumber for the buildmaster, or the buildmaster might not be running. If you see messages like 'Failure: twisted.cred.error.UnauthorizedLogin' then your worker might be using the wrong botname or password. Please correct these problems and then restart the worker. The name and password have not changed. Any suggestions? Thanks, David

2 1

Refleak 2.7 buildbot jobs are gone
by Victor Stinner 11 Feb '20

11 Feb '20

Hi, Since the Python 2.7 branch no longer accept commits, I disabled the Refleak 2.7 jobs: https://github.com/python/buildmaster-config/commit/67468e1fca8d6df8dd5215c… Regular buildbot jobs triggered on commits into the 2.7 are still enabled, but there is no more commit in this branch :-) Victor -- Night gathers, and now my watch begins. It shall not end until my death.

2 2

Additon of PPC64LE workers to the CI fleet
by Charalampos Stratakis 10 Feb '20

10 Feb '20

Hi folks, Thanks to the awesome people managing the the OpenPower hub in the Faculty of Information Technology in Brno, some ppc64le machines have been reserved for CPython's buildbot fleet. Four machines, each with 8 vCores, 8GB of RAM and 50GB of storage, running Fedora rawhide, Fedora stable, RHEL7 and RHEL8 respectively. I've already configured the workers, all that remain are the upstream bits. PR: https://github.com/python/buildmaster-config/pull/164 Note: After this change is deployed, the edelsohn-fedora-ppc64le buildbot from the gcc server farm can be decommissioned -- Regards, Charalampos Stratakis Software Engineer Python Maintenance Team, Red Hat

2 1

Two failing buildbots in cpython master
by Kyle Stanley 08 Feb '20

08 Feb '20

The following buildbots have been consistently failing across unrelated commits in cpython's master branch, during their compilation stages: "AMD64 Windows7 SP1 3.x" (https://buildbot.python.org/all/#/builders/81) and "x86 Windows7 3.x" (https://buildbot.python.org/all/#/builders/150). Based on their logs, they both seem to have been failing consistently for the last 10 days. Was there some regression introduced between the last successful build ( https://github.com/python/cpython/commit/0cd5bff6b7da3118d0c5a88fc2b80f80eb…) to the first failing build ( https://github.com/python/cpython/commit/6a65eba44bfd82ccc8bed4b5c6dd663754…), or is it unrelated? If it's unrelated, can we temporarily remove those two buildbots from the post-commit hooks to cpython's master branch until the issue is resolved?

2 2

Disk is full on buildbot server
by Victor Stinner 06 Feb '20

06 Feb '20

Hi, I had to clear old journalctl logs to free some space on buildbot: the disk was full and so Twisted failed in a database error. I had to kill the stuck process with SIGKILL. I restarted the process, but the problem will come back back soon. We should either stop to store large XML files, remove them quicker, or give more disk space to this server. Victor -- Night gathers, and now my watch begins. It shall not end until my death.

1 0