I have another edge case, and no idea what could be causing it.
In my Pyng setup, I run quite a few commands, and today I discovered
that one particular configuration just ... disappears.
The command executes once, exits, results are processed and command is
scheduled for re-execution later. But then it just ... hangs. Here is
some output which hopefully shows what happens (and I have no idea why...)
<output>
will run a1ece83c0a
ran a1ece83c0a
exited!
ended!
got results for a1ece83c0a
will run a1ece83c0a
ran a1ece83c0a
exited!
</output>
a1ece83c0a is just a check id, it is assigned to a monitor which does
periodic HTTP checks of the Exim build farm server, which I operate.
Last days I was trying to doing some front end, and now I have graphs,
you may look at the check: http://uptime.is/mon/check/a1ece83c0a
Now, note that it said "exited!" above and just would hang there. If I
hit Ctrl-C, it would break the hang:
<output>
will run a1ece83c0a
ran a1ece83c0a
exited!
ended!
got results for a1ece83c0a
will run a1ece83c0a
ran a1ece83c0a
exited!
*^Cended!
got results for a1ece83c0a*
</output>
Let me show some surrounding code:
here check is getting scheduled:
d = Deferred()
if check.id == 'a1ece83c0a':
print('will run a1ece83c0a')
check.run(d)
if check.id == 'a1ece83c0a':
print('ran a1ece83c0a')
d.addCallback(self._check_result)
d.addErrback(lambda err: self.log.error('{failure}',
failure=err))
d.addErrback(print)
in run, there's a call to spawnProcess. deferred from the scheduler is
named callback there:
d = Deferred()
proc = reactor.spawnProcess(
PyngProcess(d, input=self.input, timeout=self.run_timeout),
CMD.get(cmd[0]),
args=cmd,
env=env,
path=self.tmpdir.name if self.use_tmp and self.tmpdir
is not None else None,
)
d.addCallback(self._result, start, chain, callback)
d.addErrback(self._error, start, callback)
d.addErrback(print)
and PyngProcess is simple enough:
def processExited(self, reason):
print('exited!')
self.setTimeout(None)
def processEnded(self, reason):
print('ended!')
e = reason.value
resp = {
'stdout': self.outBuf.getvalue(),
'stderr': self.errBuf.getvalue(),
'timeout': True if self.timedOut else False,
'exit': e.exitCode,
'signal': e.signal,
'status': e.status,
}
self.deferred.callback(resp)
Now, why would this particular command have such behaviour. There's
nothing special about it, and it is running fine once. spawnProcess
would get this command and arguments:
cmd: /usr/local/bin/curl
args: ['curl', '-q', '-g', '-f', '-s', '-S', '-i', '--proto',
'=http,https', '--proto-default', 'https', '--connect-timeout', '15',
'--max-time', '30', '--retry', '0', '--retry-max-time', '30',
'--no-keepalive', '--disallow-username-in-url', '--no-location',
'--max-redirs', '0', '-6', '--user-agent', 'HTTP/pyng monitor', '--url',
'https://buildfarm.krot.org/cgi-bin/show_status.pl']
Any ideas?
I will appreciate all feedback!
-- Kirill
