Ok, looked at the logs and couldn't see anything obvious going on - tried rebooting the machine and now I can't connect to it - I'm on a trip and won't have physical access until mid-August, so it may be unavailable until then.

m

On Tue, Jul 07, 2020 at 12:45:49PM +0200, Victor Stinner wrote:

Hi Matt,

Yes, there are humans on the other side of the list :-)

I'm not aware of any action done on the buildbot server recently.

All builds of the "x86-64 macOS 3.x" worker failed with "retry lost connection" for 3 days: https://buildbot.python.org/all/#/builders/275

I suggest you to investigate on the worker side, since other workers look fine.

On the server, I found these logs:

2020-07-07 01:43:19+0000 [Broker,110,67.169.78.30] Got exception in RepeatedActionHandler Traceback (most recent call last): File "/data/buildbot/venv/lib/python3.8/site-packages/twisted/internet/defer.py", line 1475, in gotResult _inlineCallbacks(r, g, status) File "/data/buildbot/venv/lib/python3.8/site-packages/twisted/internet/defer.py", line 1416, in _inlineCallbacks result = result.throwExceptionIntoGenerator(g) File "/data/buildbot/venv/lib/python3.8/site-packages/twisted/python/failure.py", line 512, in throwExceptionIntoGenerator return g.throw(self.type, self.value, self.tb) File "/data/buildbot/venv/lib/python3.8/site-packages/buildbot/util/deferwaiter.py", line 96, in _do_action log.err(e, 'Got exception in RepeatedActionHandler') --- <exception caught here> --- File "/data/buildbot/venv/lib/python3.8/site-packages/buildbot/util/deferwaiter.py", line 94, in _do_action yield self._action() twisted.spread.pb.PBConnectionLost: [Failure instance: Traceback (failure with no frames): : Connection to the other side was lost in a non-clean fashion. ]

2020-07-07 01:43:19+0000 [Broker,110,67.169.78.30] .lostRemote 2020-07-07 01:43:19+0000 [Broker,110,67.169.78.30] Worker.detached(billenstein-macos) 2020-07-07 01:43:19+0000 [Broker,110,67.169.78.30] releaseLocks(): [] 2020-07-07 01:43:19+0000 [Broker,110,67.169.78.30] .lostRemote 2020-07-07 01:43:19+0000 [Broker,110,67.169.78.30] .lostRemote 2020-07-07 01:43:19+0000 [Broker,110,67.169.78.30] <Build x86-64 macOS custom number:31 results:success>.lostRemote 2020-07-07 01:43:19+0000 [Broker,110,67.169.78.30] stopping currentStep Test(command=['make', 'buildbottest', 'TESTOPTS=-j2 --junit-xml test-results.xml ${BUILDBOT_TESTOPTS}', 'TESTPYTHONOPTS=', 'TESTTIMEOUT=900'], timeout=1200, usePTY=False, env={}) 2020-07-07 01:43:19+0000 [Broker,110,67.169.78.30] RemoteCommand.interrupt [Failure instance: Traceback (failure with no frames):

---

Victor

Le mar. 7 juil. 2020 � 06:05, Matt Billenstein via Buildbot-status buildbot-status@python.org a �crit :

...

Anyone monitoring this email address?

The worker is running - not sure where these disconnects are coming from - is something going on with the master?

m

On Tue, Jul 07, 2020 at 02:43:19AM +0000, buildbot@python.org wrote:

...

The Buildbot working for 'Python' has noticed that the worker named billenstein-macos went away.

It last disconnected at Tue Jul 7 01:43:19 2020.

The admin on record (as reported by WORKER:info/admin) was Matt Billenstein matt@vazor.com .

Sincerely, -The Buildbot

---

Buildbot-status mailing list -- buildbot-status@python.org To unsubscribe send an email to buildbot-status-leave@python.org https://mail.python.org/mailman3/lists/buildbot-status.python.org/ Member address: vstinner@python.org

-- Night gathers, and now my watch begins. It shall not end until my death.

I've checked disk; we still have a couple hundred MB to play with, so I don't think it's related.

On Tue, Jul 7, 2020 at 11:31 AM Victor Stinner vstinner@python.org wrote:

There is an on-going problem of low disk space which might be related.

The buildbot server only has a disk of 8 GB which is full most of the time. Sadly, we failed to get a new machine with a larger disk.

I would suggest to stop storing large XML files which fill the disk.

Victor

Le mar. 7 juil. 2020 à 18:02, Matt Billenstein matt@vazor.com a écrit :

...

Ok, looked at the logs and couldn't see anything obvious going on - tried rebooting the machine and now I can't connect to it - I'm on a trip and won't have physical access until mid-August, so it may be unavailable until then.

m

On Tue, Jul 07, 2020 at 12:45:49PM +0200, Victor Stinner wrote:

...

Hi Matt,

Yes, there are humans on the other side of the list :-)

I'm not aware of any action done on the buildbot server recently.

All builds of the "x86-64 macOS 3.x" worker failed with "retry lost connection" for 3 days: https://buildbot.python.org/all/#/builders/275

I suggest you to investigate on the worker side, since other workers look fine.

On the server, I found these logs:

2020-07-07 01:43:19+0000 [Broker,110,67.169.78.30] Got exception in RepeatedActionHandler Traceback (most recent call last): File "/data/buildbot/venv/lib/python3.8/site-packages/twisted/internet/defer.py", line 1475, in gotResult _inlineCallbacks(r, g, status) File "/data/buildbot/venv/lib/python3.8/site-packages/twisted/internet/defer.py", line 1416, in _inlineCallbacks result = result.throwExceptionIntoGenerator(g) File "/data/buildbot/venv/lib/python3.8/site-packages/twisted/python/failure.py", line 512, in throwExceptionIntoGenerator return g.throw(self.type, self.value, self.tb) File "/data/buildbot/venv/lib/python3.8/site-packages/buildbot/util/deferwaiter.py", line 96, in _do_action log.err(e, 'Got exception in RepeatedActionHandler') --- <exception caught here> --- File "/data/buildbot/venv/lib/python3.8/site-packages/buildbot/util/deferwaiter.py", line 94, in _do_action yield self._action() twisted.spread.pb.PBConnectionLost: [Failure instance: Traceback (failure with no frames): : Connection to the other side was lost in a non-clean fashion. ]

2020-07-07 01:43:19+0000 [Broker,110,67.169.78.30] .lostRemote 2020-07-07 01:43:19+0000 [Broker,110,67.169.78.30] Worker.detached(billenstein-macos) 2020-07-07 01:43:19+0000 [Broker,110,67.169.78.30] releaseLocks(): [] 2020-07-07 01:43:19+0000 [Broker,110,67.169.78.30] .lostRemote 2020-07-07 01:43:19+0000 [Broker,110,67.169.78.30] .lostRemote 2020-07-07 01:43:19+0000 [Broker,110,67.169.78.30] <Build x86-64 macOS custom number:31 results:success>.lostRemote 2020-07-07 01:43:19+0000 [Broker,110,67.169.78.30] stopping currentStep Test(command=['make', 'buildbottest', 'TESTOPTS=-j2 --junit-xml test-results.xml ${BUILDBOT_TESTOPTS}', 'TESTPYTHONOPTS=', 'TESTTIMEOUT=900'], timeout=1200, usePTY=False, env={}) 2020-07-07 01:43:19+0000 [Broker,110,67.169.78.30] RemoteCommand.interrupt [Failure instance: Traceback (failure with no frames):

---

Victor

Le mar. 7 juil. 2020 � 06:05, Matt Billenstein via Buildbot-status buildbot-status@python.org a �crit :

...

Anyone monitoring this email address?

The worker is running - not sure where these disconnects are coming from - is something going on with the master?

m

On Tue, Jul 07, 2020 at 02:43:19AM +0000, buildbot@python.org wrote:

...

The Buildbot working for 'Python' has noticed that the worker named billenstein-macos went away.

It last disconnected at Tue Jul 7 01:43:19 2020.

The admin on record (as reported by WORKER:info/admin) was Matt Billenstein matt@vazor.com .

Sincerely, -The Buildbot

---

Buildbot-status mailing list -- buildbot-status@python.org To unsubscribe send an email to buildbot-status-leave@python.org https://mail.python.org/mailman3/lists/buildbot-status.python.org/ Member address: vstinner@python.org

-- Night gathers, and now my watch begins. It shall not end until my death.

-- Night gathers, and now my watch begins. It shall not end until my death.

Where is the machine hosted? If it’s ec2 ebs, fairly painless to stop the machine, expand the volume, then extend it using e2resize.

M

-- Matt Billenstein matt@vazor.com

On Jul 7, 2020, at 9:41 AM, Zachary Ware zach@python.org wrote:

I've checked disk; we still have a couple hundred MB to play with, so I don't think it's related.

...

On Tue, Jul 7, 2020 at 11:31 AM Victor Stinner vstinner@python.org wrote:

There is an on-going problem of low disk space which might be related.

The buildbot server only has a disk of 8 GB which is full most of the time. Sadly, we failed to get a new machine with a larger disk.

I would suggest to stop storing large XML files which fill the disk.

Victor

...

Le mar. 7 juil. 2020 à 18:02, Matt Billenstein matt@vazor.com a écrit :

Ok, looked at the logs and couldn't see anything obvious going on - tried rebooting the machine and now I can't connect to it - I'm on a trip and won't have physical access until mid-August, so it may be unavailable until then.

m

On Tue, Jul 07, 2020 at 12:45:49PM +0200, Victor Stinner wrote:

...

Hi Matt,

Yes, there are humans on the other side of the list :-)

I'm not aware of any action done on the buildbot server recently.

All builds of the "x86-64 macOS 3.x" worker failed with "retry lost connection" for 3 days: https://buildbot.python.org/all/#/builders/275

I suggest you to investigate on the worker side, since other workers look fine.

On the server, I found these logs:

2020-07-07 01:43:19+0000 [Broker,110,67.169.78.30] Got exception in RepeatedActionHandler Traceback (most recent call last): File "/data/buildbot/venv/lib/python3.8/site-packages/twisted/internet/defer.py", line 1475, in gotResult _inlineCallbacks(r, g, status) File "/data/buildbot/venv/lib/python3.8/site-packages/twisted/internet/defer.py", line 1416, in _inlineCallbacks result = result.throwExceptionIntoGenerator(g) File "/data/buildbot/venv/lib/python3.8/site-packages/twisted/python/failure.py", line 512, in throwExceptionIntoGenerator return g.throw(self.type, self.value, self.tb) File "/data/buildbot/venv/lib/python3.8/site-packages/buildbot/util/deferwaiter.py", line 96, in _do_action log.err(e, 'Got exception in RepeatedActionHandler') --- <exception caught here> --- File "/data/buildbot/venv/lib/python3.8/site-packages/buildbot/util/deferwaiter.py", line 94, in _do_action yield self._action() twisted.spread.pb.PBConnectionLost: [Failure instance: Traceback (failure with no frames): : Connection to the other side was lost in a non-clean fashion. ]

2020-07-07 01:43:19+0000 [Broker,110,67.169.78.30] .lostRemote 2020-07-07 01:43:19+0000 [Broker,110,67.169.78.30] Worker.detached(billenstein-macos) 2020-07-07 01:43:19+0000 [Broker,110,67.169.78.30] releaseLocks(): [] 2020-07-07 01:43:19+0000 [Broker,110,67.169.78.30] .lostRemote 2020-07-07 01:43:19+0000 [Broker,110,67.169.78.30] .lostRemote 2020-07-07 01:43:19+0000 [Broker,110,67.169.78.30] <Build x86-64 macOS custom number:31 results:success>.lostRemote 2020-07-07 01:43:19+0000 [Broker,110,67.169.78.30] stopping currentStep Test(command=['make', 'buildbottest', 'TESTOPTS=-j2 --junit-xml test-results.xml ${BUILDBOT_TESTOPTS}', 'TESTPYTHONOPTS=', 'TESTTIMEOUT=900'], timeout=1200, usePTY=False, env={}) 2020-07-07 01:43:19+0000 [Broker,110,67.169.78.30] RemoteCommand.interrupt [Failure instance: Traceback (failure with no frames):

---

Victor

Le mar. 7 juil. 2020 � 06:05, Matt Billenstein via Buildbot-status buildbot-status@python.org a �crit :

...

Anyone monitoring this email address?

The worker is running - not sure where these disconnects are coming from - is something going on with the master?

m

On Tue, Jul 07, 2020 at 02:43:19AM +0000, buildbot@python.org wrote:

...

The Buildbot working for 'Python' has noticed that the worker named billenstein-macos went away.

It last disconnected at Tue Jul 7 01:43:19 2020.

The admin on record (as reported by WORKER:info/admin) was Matt Billenstein matt@vazor.com .

Sincerely, -The Buildbot

---

Buildbot-status mailing list -- buildbot-status@python.org To unsubscribe send an email to buildbot-status-leave@python.org https://mail.python.org/mailman3/lists/buildbot-status.python.org/ Member address: vstinner@python.org

-- Night gathers, and now my watch begins. It shall not end until my death.

-- Night gathers, and now my watch begins. It shall not end until my death.