[Twisted-Python] deferring result to PB a callRemote method
I am writing a tool for work that will run various software development aids such as message capturing and diagnostic control of system processes. My current design strategy is to implement these aids as plugins (not twisted plugins) to a generic plugin runner rather than stand alone applications. The plugins are spawned as separate processes by the plugin-runner and communicate using Perspective Broker during the plugin installation and shutdown phases. Some plugins may take a while to shutdown as they need to close sockets, finalize files, move large files, etc. I would like to instruct the plugin to shutdown and be told when it is finally ready to be shut down. To date all my callRemote methods have effectively returned immediately. For example: def remote_get_name(self): return self.name My question to the list was going to be: Is there a pattern/example I could follow where I could call plugin.callRemote("shutdown") on a plugin and not return a result to the plugin-runner until the plugin has completed all it's, potentially, long running activities? However, while I was trying to write a small code snippet that would demonstrate what I was wanting, I think I got it working. I think the simple answer to my question is to just return a deferred as the result to the callRemote("shutdown") method and trigger it as normal. Google is my friend but I could not find examples of this usage. Is there any references to this usage in the twisted docs? Below (and attached in case the formatting goes screwy) is a short example which emulates a long running shutdown activity performed by the plugin prior to shutdown. It seems to delay the processing of the callRemote("shutdown") result until the plugin has completed it's long running activity. I have omitted the separate process stuff as it didn't seem relevant for this snippet. Is the following code snippet the standard/normal way to defer the return result of a callRemote method call? If this is the normal way, how does triggering the deferred on the plugin (client) side also trigger the same/copy deferred returned to the plugin-runner (server)? Is this PB magic, somehow managing deferreds across the PB interface? Regards, Chris from twisted.internet import reactor, defer from twisted.spread import pb import datetime class PluginClient(pb.Referenceable): """ Plugin client interface exposed to PluginServer's """ def __init__(self, shutdownCallback): self.shutdownHandler = shutdownCallback def remote_shutdown(self): """ Instruct Plugin to shutdown """ print "plugin instructed to shutdown" d = defer.Deferred() self.shutdownHandler(d) return d class PluginServer(pb.Root): """ Plugin server interface exposed to PluginClient's """ def remote_set_client_perspective(self, pluginRef): print "plugin-runner got client reference" reactor.callLater(1.0, self.shutdown_plugin, pluginRef) def shutdown_plugin(self, pluginRef): def pluginShutdownCompleted(result, startTime): endTime = datetime.datetime.now() print "Plugin shutdown took %s to complete." % (endTime - startTime) return result print "plugin-runner asking plugin to shutdown" d = pluginRef.callRemote("shutdown") d.addCallback(pluginShutdownCompleted, datetime.datetime.now()) d.addCallback(self.shutdown) def shutdown(self, _): reactor.stop() def startPluginServer(port): """ Start a plugin communications server """ print "starting server" reactor.listenTCP(port=port, factory=pb.PBServerFactory(PluginServer()), interface='localhost') def startPluginClient(port, shutdownHandler): """ Start a plugin communications client """ def gotServerPerspective(serverPerspective, pluginPerspective): """ Give the plugin-runner this client's perspective """ serverPerspective.callRemote("set_client_perspective", pluginPerspective) return serverPerspective print "starting plugin" client = PluginClient(shutdownHandler) factory = pb.PBClientFactory() reactor.connectTCP(host='localhost', port=port, factory=factory) return factory.getRootObject().addCallback(gotServerPerspective, client) if __name__ == "__main__": port = 42155 def longRunningAction(d, countDown=10): """ Emulate long running shutdown activities """ print "shuting down in %i seconds" % countDown if countDown == 0: d.callback(True) else: countDown -= 1 reactor.callLater(1, longRunningAction, d, countDown) # start plugin-runner reactor.callWhenRunning(startPluginServer, port) # start plugin reactor.callLater(2.0, startPluginClient, port, longRunningAction) reactor.run()
Chris Laws wrote: [...]
I think the simple answer to my question is to just return a deferred as the result to the callRemote("shutdown")�method and trigger it as normal.
Yes, that's right. You can return Deferreds from your PB server's methods. So long as the eventual result is something serializable over PB (e.g. a Referenceable or a Copyable), it will work.
Google is my friend but I could not find examples of this usage. Is there any references to this usage in the twisted docs?
Huh, surprisingly not in the primary PB docs. The Twisted “finger” tutorial does do this, though, if you read it carefully enough. See finger21.tac in http://twistedmatrix.com/documents/current/core/howto/tutorial/pb.html; the remote_* methods of PerspectiveFingerFromService delegate to self.service, which is FingerService, which returns Deferreds from its getUser and getUsers methods. [...]
Is the following code snippet the standard/normal way to defer the return result of a callRemote method call?
It's a bit odd. e.g. You do this: def remote_shutdown(self): """ Instruct Plugin to shutdown """ print "plugin instructed to shutdown" d = defer.Deferred() self.shutdownHandler(d) return d This can work, so long as on what self.shutdownHandler calls d.callback/errback, as yours does. But typically it would be written as: def remote_shutdown(self): """ Instruct Plugin to shutdown """ print "plugin instructed to shutdown" return self.shutdownHandler() i.e. typically the code that is calling d.callback is also responsible for creating d.
If this is the normal way, how does triggering the deferred on the plugin (client) side also trigger the same/copy deferred returned to the plugin-runner (server)? Is this PB magic, somehow managing deferreds across the PB interface?
PB doesn't actually send the Deferred over the network, but it does manage the deferred for you. It works because when you return a Deferred, PB on the server will add a callback to it that will send that eventual result over the network, rather than sending one immediately. On the client side, as the PB docs point out: “because of the delay involved [waiting for a network connection], callRemote() returns a Deferred.” (From http://twistedmatrix.com/documents/current/core/howto/pb-usage.html). The client doesn't know or care if the server used a Deferred or not, the bytes on the wire are the same. (And a delay caused by the server waiting for a Deferred to fire could just as easily be a delay caused by network congestion.) I wouldn't describe it as “magic” so much as “a really convenient API”. -Andrew.
On Mon, Jan 18, 2010 at 7:19 PM, Andrew Bennetts
Google is my friend but I could not find examples of this usage. Is there any references to this usage in the twisted docs?
Huh, surprisingly not in the primary PB docs. The Twisted “finger” tutorial does do this, though, if you read it carefully enough. See finger21.tac in http://twistedmatrix.com/documents/current/core/howto/tutorial/pb.html; the remote_* methods of PerspectiveFingerFromService delegate to self.service, which is FingerService, which returns Deferreds from its getUser and getUsers methods.
It sems like this should be discussed in the PB docs. Maybe someone should file a ticket.. (hint, hint) Kevin Horn
participants (3)
-
Andrew Bennetts -
Chris Laws -
Kevin Horn