[Python-Dev] funky buildbot

[Bill Janssen]

My Intel Snow Leopard buildbot failed again last night.

Here's what I see in twistd.log.  Any of this look familiar to some one?
Failed at about 19:58 (PST?).  I don't see any reason for these
failures, but they seem to happen about once a week.  This is the
standard Snow Leopard Python 2.6.1, and whatever version of Twisted
comes with Snow Leopard, and the latest buildslave.  The machine is in a
server room with conditioned power supply and pretty good network
access.  The two other PPC machines next to it have no problems, but
they're running older versions of OS X.

Bill

[...]
2011-03-22 19:44:07-0700 [Broker,client] RunProcess._startCommand
2011-03-22 19:44:07-0700 [Broker,client]  make buildbottest TESTOPTS= TESTPYTHONOPTS=
2011-03-22 19:44:07-0700 [Broker,client]   in dir /Users/buildbot/buildarea/3.x.parc-snowleopard-1/build (timeout 3600 secs)
2011-03-22 19:44:07-0700 [Broker,client]   watching logfiles {}
2011-03-22 19:44:07-0700 [Broker,client]   argv: ['make', 'buildbottest', 'TESTOPTS=', 'TESTPYTHONOPTS=']
2011-03-22 19:44:07-0700 [Broker,client]  environment: {'LANG': 'en_US.UTF-8', 'TERM': 'xterm-color', 'Apple_PubSub_Socket_Render': '/tmp/launch-c3R8Zl/Render', 'TMPDIR': '/var/folders/3M/3MMiDEqmEiGyT5q5zPW31++++TM/-Tmp-/', 'VERSIONER_PYTHON_VERSION': '2.6', 'SHLVL': '1', 'VERSIONER_PYTHON_PREFER_32_BIT': 'no', 'SSH_AUTH_SOCK': '/tmp/launch-Nl3TOy/Listeners', '__CF_USER_TEXT_ENCODING': '0x1F6:0:0', 'PWD': '/Users/buildbot/buildarea/3.x.parc-snowleopard-1/build', 'SHELL': '/bin/bash', 'LOGNAME': 'buildbot', 'TERM_PROGRAM_VERSION': '273', 'PATH': '/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/X11/bin', 'HOME': '/Users/buildbot', 'TERM_PROGRAM': 'Apple_Terminal', 'COMMAND_MODE': 'unix2003', 'DISPLAY': '/tmp/launch-iaSAFg/org.x:0', '_': '/usr/local/bin/buildslave', 'USER': 'buildbot'}
2011-03-22 19:44:07-0700 [Broker,client]   closing stdin
2011-03-22 19:44:07-0700 [Broker,client]   using PTY: False
2011-03-22 19:46:17-0700 [-] sending app-level keepalive
2011-03-22 19:56:17-0700 [-] sending app-level keepalive
2011-03-22 19:58:33-0700 [Broker,client] SlaveBuilder._ackFailed: SlaveBuilder.sendUpdate
2011-03-22 19:58:33-0700 [Broker,client] Unhandled Error
	Traceback (most recent call last):
	Failure: twisted.spread.pb.PBConnectionLost: [Failure instance: Traceback (failure with no frames): <class 'twisted.internet.error.ConnectionLost'>: Connection to the other side was lost in a non-clean fashion.
	]
	
2011-03-22 19:58:33-0700 [Broker,client] lost remote
2011-03-22 19:58:33-0700 [Broker,client] lost remote
2011-03-22 19:58:33-0700 [Broker,client] lost remote
2011-03-22 19:58:33-0700 [Broker,client] lost remote
2011-03-22 19:58:33-0700 [Broker,client] lost remote step
2011-03-22 19:58:33-0700 [Broker,client] stopCommand: halting current command <buildslave.commands.shell.SlaveShellCommand instance at 0x102002c68>
2011-03-22 19:58:33-0700 [Broker,client] command interrupted, killing pid 90085
2011-03-22 19:58:33-0700 [Broker,client] trying os.kill(-pid, 9)
2011-03-22 19:58:33-0700 [Broker,client] trying process.signalProcess('KILL')
2011-03-22 19:58:33-0700 [Broker,client]  signal KILL sent successfully
2011-03-22 19:58:33-0700 [Broker,client] Lost connection to dinsdale.python.org:9020
2011-03-22 19:58:33-0700 [Broker,client] <twisted.internet.tcp.Connector instance at 0x101629a70> will retry in 2 seconds
2011-03-22 19:58:33-0700 [Broker,client] Stopping factory <buildslave.bot.BotFactory instance at 0x101629998>
2011-03-22 19:58:33-0700 [-] command finished with signal 9, exit code None, elapsedTime: 865.868813
2011-03-22 19:58:33-0700 [-] SlaveBuilder.commandComplete None
2011-03-22 19:58:35-0700 [-] Starting factory <buildslave.bot.BotFactory instance at 0x101629998>
2011-03-22 19:58:35-0700 [-] Connecting to dinsdale.python.org:9020
2011-03-22 19:58:36-0700 [Uninitialized] Connection to dinsdale.python.org:9020 failed: Connection Refused
2011-03-22 19:58:36-0700 [Uninitialized] <twisted.internet.tcp.Connector instance at 0x101629a70> will retry in 7 seconds
2011-03-22 19:58:36-0700 [Uninitialized] Stopping factory <buildslave.bot.BotFactory instance at 0x101629998>
2011-03-22 19:58:43-0700 [-] Starting factory <buildslave.bot.BotFactory instance at 0x101629998>
2011-03-22 19:58:43-0700 [-] Connecting to dinsdale.python.org:9020