[Twisted-Python] Crash when using XmlRPC during reactor shutdown
Hi, I have been experiencing a crash when trying to use the XmlRPC query subsystem while the reactor is shutting down. The context of it is that my twisted application talks to an upstream server using XmlRPC. When it comes online, it registers, and when it goes offline, it should unregister itself, using a simple XmlRPC query. So, I figured the natural place to put the notifications were in the startService and stopService methods, that get called at just the right time. startService is not a problem, and works fine. for stopService, I made the method return the XmlRPC Deferred, so that the reactor would wait until request completion before continuing the shutdown. The thing is, with this scheme Twisted dumps an exception during shutdown, because the XmlRPC Deferred gets fired twice. I've reproduced this behaviour with a minimal application: **************************************************************** **************************************************************** from twisted.application.service import Service from twisted.python import log from twisted.web.xmlrpc import Proxy from twisted.application.service import Application, IServiceCollection from twisted.internet import reactor, defer class TestXmlRpcService(Service): def startService(self): log.msg("Starting service") reactor.callLater(0, reactor.stop) def stopService(self): def handle_success(res): log.msg("Unregistration successful, but reactor will now crash") return True def handle_error(e): log.msg("Unregistration failed...") return False log.msg("Calling XmlRPC backend (and crashing)") p = Proxy('http://natulte.net/pub/fgs/RPC2.php') defer.setDebugging(True) # Call any old method, this is just a proof of concept d = p.callRemote('fgsd.register', "00000000000000000000000000000000", "bleh") d.addCallbacks(handle_success, handle_error) return d service = TestXmlRpcService() application = Application("proxy") service.setServiceParent(application) **************************************************************** **************************************************************** If you run that with `twistd -n -l- -y test.tac`, you should see the crash immediately. I know that patients shouldn't give diagnostics about their problems to the doctor, but nevertheless, examining the exception log, I believe I know what the problem is. - When stopService returns a Deferred, the reactor stops the shutdown and adds callbacks to the Deferred, to resume shutdown at the right time. - The XmlRPC query succeeds, and fires the Deferred. It walks right down the notification chain, and eventually trips the reactor's callback. - The reactor, within the callback, resumes shutdown, and calls disconnectAll to kill all active connections. - At this point, the XmlRPC query still has its socket open, and so gets notified that the connection was lost. It attempts to fire our Deferred's errback to notify of this, and foom, AlreadyCalledError. If this makes sense, the resolution might be simple: close the transport connection before firing the deferred, so that the reactor can't kill the connection and trigger the double-call. The other option would be to include a flag in the xmlrpc query object, to avoid double-firing in this (admitedly corner-case) situation. Does all that make sense? And, of course, does anyone have an idea of a workaround I could use in the meantime? As I'm shutting down the service anyway, the exception is not really a problem in itself, it's just a little unclean to have an exception dump in the logs at each shutdown. Thanks in advance (and already, for Twisted :-), - Dave.
* David Anderson <david.anderson@calixo.net> [2006-04-29 17:10:06]:
I have been experiencing a crash when trying to use the XmlRPC query subsystem while the reactor is shutting down.
After a bit more fiddling, I found a functional workaround. The trick is indeed to let the xmlrpc Deferred fully terminate *before* the reactor continues the shutdown, instead of continuing the shutdown within the Deferred callback. With that in mind, the new stopService makes the RPC call, but creates, stores and returns a different Deferred. The RPC Deferred gets a callback that calls reactor.callLater to schedule the firing of the reactor's Deferred in 0 seconds. That lets the xmlrpc deferred complete and clean up, and immediately after the reactor schedules its own shutdown callback, and all is well. Here is the new stopService code: **************************************************************** **************************************************************** def stopService(self): def handle_success(res): log.msg("Unregistration successful, and reactor won't crash") reactor.callLater(0, self._def.callback, True) def handle_error(e): log.msg("Unregistration failed...") reactor.callLater(0, self._def.callback, False) log.msg("Calling XmlRPC backend (and crashing)") p = Proxy('http://natulte.net/pub/fgs/RPC2.php') d = p.callRemote('fgsd.register', "00000000000000000000000000000000", "bleh") self._def = defer.Deferred() d.addCallbacks(handle_success, handle_error) return self._def **************************************************************** **************************************************************** Nevertheless, this feels more like a workaround than a complete solution. What do you all think? - Dave.
participants (1)
-
David Anderson