[Twisted-Python] Dealing with an intermittent PB server
I'm rendering the results of a remote method call:
def data_tableList(self, ctx, data): ... d = self.pbClientFactory.login(creds) d.addCallback(lambda object: object.callRemote("foo")) return d
Works, great if the PB server is up, but if it's down the browser just hangs. Is there a correct way to check that the connection is up at this point? This works:
if self.pbClientFactory._broker: d = self.pbClientFactory.login(creds) d.addCallback(lambda object: object.callRemote("getAlertReports")) else: d = [] return d
but seems like a hack. Also, ideally, I'd like to attempt a reconnection to the PB server at this point if it's not running. What's the best way to do that?
Thanks, Dave Cook
On Tue, 15 Feb 2005 19:56:15 -0800, Dave Cook daverz@gmail.com wrote:
I'm rendering the results of a remote method call:
def data_tableList(self, ctx, data): ... d = self.pbClientFactory.login(creds) d.addCallback(lambda object: object.callRemote("foo")) return d
Always always always use errbacks. *Especially* for Deferreds that can fail due to transient network conditions.
def _cbLogin(self, object): return object.callRemote("foo")
def _ebLogin(self, failure): return "Sorry, couldn't log in! (Try hitting ^R maybe)"
def _ebCallFoo(self, failure): return "Sorry, calling foo failed! (Try hitting ^R maybe)"
def data_tableList(self, ctx, data): ... d = self.pbClientFactory.login(creds) d.addCallbacks(self._cbLogin, self._ebLogin) d.addErrback(self._ebCallFoo) return d
These errbacks are lame, but they cause the page to be renderable (and more generally, they let the asynchronous operation come to some kind of logical conclusion, instead of hanging forever, waiting for an error handler which will never arrive).
Better ones could automatically retry the call, or inform an administrator of a possible misconfiguration, or some combination, or something even smarter.
[snip] but seems like a hack. Also, ideally, I'd like to attempt a reconnection to the PB server at this point if it's not running. What's the best way to do that?
clientConnectionFailed will be called on the ClientFactory, you should notice this and attempt to reconnect. Whether you fail any outstanding Deferreds which were waiting on the existing connection or let them survive to possibly succeed over the new one is up to you.
Jp
On Tue, 15 Feb 2005 20:29:20 -0800 (PST), Jp Calderone exarkun@divmod.com wrote:
Always always always use errbacks. *Especially* for Deferreds that can fail due to transient network conditions.
Thanks. _ebLogin never seems to get called, though. I can see that tne PBClientFactory is stopped, so I assume that trying to login should trigger an error.
Dave Cook
On Feb 15, 2005, at 11:28 PM, Jp Calderone wrote:
On Tue, 15 Feb 2005 19:56:15 -0800, Dave Cook daverz@gmail.com wrote: Always always always use errbacks. *Especially* for Deferreds that can fail due to transient network conditions.
Really that advice applies to Nevow too. *It* should be adding an errback to that deferred to do something useful like aborting the connection or printing an error.
James
On Wed, 16 Feb 2005 11:56:20 -0500, James Y Knight foom@fuhm.net wrote:
On Feb 15, 2005, at 11:28 PM, Jp Calderone wrote:
On Tue, 15 Feb 2005 19:56:15 -0800, Dave Cook daverz@gmail.com wrote: Always always always use errbacks. *Especially* for Deferreds that can fail due to transient network conditions.
Really that advice applies to Nevow too. *It* should be adding an errback to that deferred to do something useful like aborting the connection or printing an error.
James
Good point. If there's no way the Deferred can get back to application code from the framework, then the framework should take responsibility for error handling. This is the same as using a try/bare-except in framework code around synchronous calls into application code.
Jp
On Wed, Feb 16, 2005 at 04:28:32AM +0000, Jp Calderone wrote:
On Tue, 15 Feb 2005 19:56:15 -0800, Dave Cook daverz@gmail.com wrote:
I'm rendering the results of a remote method call:
def data_tableList(self, ctx, data): ... d = self.pbClientFactory.login(creds) d.addCallback(lambda object: object.callRemote("foo")) return dAlways always always use errbacks. *Especially* for Deferreds that can fail due to transient network conditions.
See, that depends. I ran into this just today in fact, and found that if the server has gone away, then you'll get an exception *right then*, not via the errback:
def do_stop(self): try: self.root.callRemote('stop', self).addCallbacks(self.stop, self.err) except pb.DeadReferenceError, re: self.stale()
Why, why, why why why doesn't the DeadReferenceError go down the errback? As it is, the code has to be littered with both synchronous and asynchronous error handling, which is needlessly irritating, surely?
The original poster may find his problem was a DeadReferenceError that's getting silently eaten (or just not GCed right away) by the framework, as I did.
For the original poster, here's what I've got:
class Thing: def __init__(self): self.root = None self.connect()
def connect(self): f = pb.PBClientFactory() reactor.connectTCP(host, port, f) f.getRootObject().addCallbacks(self.connect_ok, self.connect_fail)
def connect_ok(self, root): self.root = root
def connect_fail(self, f): self.banner(f.getErrorMessage()) reactor.callLater(1, self.connect)
def stale(self): self.root = None self.connect()
def start(self): if not self.root: self.banner("Connection in progress, please wait") return
try: self.root.callRemote('start').addCallbacks(self.start2, self.err) self.banner("sent...") except pb.DeadReferenceError, re: self.stale() self.banner("Connection dropped, reconnecting")
def start2(self, dummy): self.banner("started")
def err(self, f): self.banner(f.getErrorMessage())
On Thu, 17 Feb 2005 01:05:22 +0000, Phil Mayers p.mayers@imperial.ac.uk wrote:
On Wed, Feb 16, 2005 at 04:28:32AM +0000, Jp Calderone wrote:
On Tue, 15 Feb 2005 19:56:15 -0800, Dave Cook daverz@gmail.com wrote:
I'm rendering the results of a remote method call:
def data_tableList(self, ctx, data): ... d = self.pbClientFactory.login(creds) d.addCallback(lambda object: object.callRemote("foo")) return dAlways always always use errbacks. *Especially* for Deferreds that can fail due to transient network conditions.
See, that depends. I ran into this just today in fact, and found that if the server has gone away, then you'll get an exception *right then*, not via the errback:
def do_stop(self): try: self.root.callRemote('stop', self).addCallbacks(self.stop, self.err) except pb.DeadReferenceError, re: self.stale()
Why, why, why why why doesn't the DeadReferenceError go down the errback? As it is, the code has to be littered with both synchronous and asynchronous error handling, which is needlessly irritating, surely?
Yes, this is unfortunate. Changing it will break programs that handle the error as it currently occurs, though. Brian Warner has been working on an improved version of PB for some time. It is mostly API compatible, and does a lot fewer irritating things of this sort.
The original poster may find his problem was a DeadReferenceError that's getting silently eaten (or just not GCed right away) by the framework, as I did.
Not quite. Twisted doesn't silently eat failures. But if you forget to add an errback, it might seem that way :) So this should just be taken as emphasis on my original point.
If there _are_ framework pieces that are supposed to become responsible for a Deferred's error handling but aren't setting up errbacks, that's a bug. If you know of any, please report them on the bug tracker. (The Nevow example James Knight pointed out is probably one such case.)
Jp
On Thu, 17 Feb 2005 01:05:22 +0000, Phil Mayers p.mayers@imperial.ac.uk wrote:
try: self.root.callRemote('start').addCallbacks(self.start2, self.err)
I never get this far. In my code:
def data_tableList(self, ctx, data): ... d = self.pbClientFactory.login(creds).addCallbacks(self._cbLogin, self._ebLogin) return d
Neither _cbLogin or _ebLogin ever get called, so there's no possiblity of the DeadReferenceError getting called. I guess this could be a Nevow bug (I hate bugging people if it's just my misunderstanding)? I think what I'll do is keep a reference to the original SSLClient object and check client.running before I try factory.login().
On Thu, 2005-02-17 at 11:31 -0800, Dave Cook wrote:
Neither _cbLogin or _ebLogin ever get called, so there's no possiblity of the DeadReferenceError getting called. I guess this could be a Nevow bug (I hate bugging people if it's just my misunderstanding)? I think what I'll do is keep a reference to the original SSLClient object and check client.running before I try factory.login().
Are you sure the SSLClient is being told to connect? I.e. it is hooked up to an Application instance run by twistd, or has startService called explicitly by you?
On Thu, 17 Feb 2005 15:30:38 -0500, Itamar Shtull-Trauring itamar@itamarst.org wrote:
Are you sure the SSLClient is being told to connect? I.e. it is hooked up to an Application instance run by twistd, or has startService called explicitly by you?
Yes, because when I make sure the corresponding SSLServer is running first, rendering the data from the remote call works fine. But I'm also testing with the SSLServer down.
Dave Cook
On Thu, Feb 17, 2005 at 11:31:24AM -0800, Dave Cook wrote:
On Thu, 17 Feb 2005 01:05:22 +0000, Phil Mayers p.mayers@imperial.ac.uk wrote:
try: self.root.callRemote('start').addCallbacks(self.start2, self.err)I never get this far. In my code:
def data_tableList(self, ctx, data): ... d = self.pbClientFactory.login(creds).addCallbacks(self._cbLogin, self._ebLogin) return d
Neither _cbLogin or _ebLogin ever get called, so there's no possiblity of the DeadReferenceError getting called. I guess this could be a
That's odd. Even if Nevow discarded the deferred (not sure - I don't use it in data/render mode, my app is 100% livepage!) the callback/errback should still get called by the reactor.
If I call "start" in my example with the server down, I get a Failure to errback with "errno 111 connection refused" (I see it in the LivePage, via my "alert" function)
Is there any possibility there's a firewall between the Nevow middle and PB backend that's making TCP RST or ICMPs not work?
Nevow bug (I hate bugging people if it's just my misunderstanding)? I think what I'll do is keep a reference to the original SSLClient object and check client.running before I try factory.login().
Wait, SSL? Hmm. Have you tried running over straight TCP just to see if there's a difference (I'm thinking error handling propagation etc. given the layered nature of SSL and comments seen on this list to the effect of the Twisted SSL implementation complexity)
On Thu, 17 Feb 2005 23:24:19 +0000, Phil Mayers p.mayers@imperial.ac.uk wrote:
Is there any possibility there's a firewall between the Nevow middle and PB backend that's making TCP RST or ICMPs not work?
When testing, I'm either running both server and client on the same machine or on machines on the same switch.
Wait, SSL? Hmm. Have you tried running over straight TCP just to see if there's a difference (I'm thinking error handling propagation etc. given the layered nature of SSL and comments seen on this list to the effect of the Twisted SSL implementation complexity)
I'll try working up a test case, starting without SSL or authentication, and working my way up.
Dave Cook
On Thu, 17 Feb 2005 11:31:24 -0800, Dave Cook daverz@gmail.com wrote:
On Thu, 17 Feb 2005 01:05:22 +0000, Phil Mayers p.mayers@imperial.ac.uk wrote:
try: self.root.callRemote('start').addCallbacks(self.start2, self.err)I never get this far. In my code:
def data_tableList(self, ctx, data): ... d = self.pbClientFactory.login(creds).addCallbacks(self._cbLogin,self._ebLogin) return d
Are you totally certain _cbLogin is never called? Remember, _ebLogin _will not_ be used as the errback for _cbLogin when it is added in the above manner: it will only handle errors from the .login() Deferred. To have _ebLogin handle all errors (except for those from itself):
def data_tableList(self, ctx, data): ... d = self.pbClientFactory.login(creds) d.addCallback(self._cbLogin) d.addErrback(self._ebLogin) return d
Or, to have _ebLogin continue handling errors from .login() and to have a separate error handler for _cbLogin and _ebLogin():
def data_tableList(self, ctx, data): ... d = self.pbClientFactory.login(creds) d.addCallbacks(self._cbLogin, self._ebLogin) d.addErrback(self._ebCallbacks) return d
twisted.python.log.err is a really handy thing to use in place of self._ebCallbacks while debugging. It can very quickly show you an error you didn't even realize was happening (often a silly one - like declaring the wrong number of arguments to a callback function, causing a TypeError when Twisted tries to call it, making it appear as though the function is never called at all).
Jp
On Thu, 17 Feb 2005 01:05:22 +0000, Phil Mayers p.mayers@imperial.ac.uk wrote:
def connect(self): f = pb.PBClientFactory() reactor.connectTCP(host, port, f) f.getRootObject().addCallbacks(self.connect_ok, self.connect_fail)
Maybe I should do it the way you do here and not create the client factory until I need it rather than at the start of my application. How costly is it to create the client factories each time?
Dave Cook
On Thu, Feb 17, 2005 at 02:51:28PM -0800, Dave Cook wrote:
On Thu, 17 Feb 2005 01:05:22 +0000, Phil Mayers p.mayers@imperial.ac.uk wrote:
def connect(self): f = pb.PBClientFactory() reactor.connectTCP(host, port, f) f.getRootObject().addCallbacks(self.connect_ok, self.connect_fail)
Maybe I should do it the way you do here and not create the client factory until I need it rather than at the start of my application. How costly is it to create the client factories each time?
I don't honestly know - not significantly I think, looking at the code (but then "not significantly" depends on if you're talking once a second or a hundred time a second, really :o)
On Wednesday 16 February 2005 04:56, Dave Cook wrote:
I'm rendering the results of a remote method call:
def data_tableList(self, ctx, data): ... d = self.pbClientFactory.login(creds) d.addCallback(lambda object: object.callRemote("foo")) return dbut seems like a hack. Also, ideally, I'd like to attempt a reconnection to the PB server at this point if it's not running. What's the best way to do that?
Maybe it's because the connection succeeds, but *after* that, your server crashes or something else, so it's not catched by this code. You should use the notifyOnDisconnect() method for this. Here's a snippet of code I use regularly to do this:
[...] def connect(self): factory = pb.PBClientFactory() reactor.connectTCP(distantServer, distantPort, factory) def1 = factory.login(credentials.UsernamePassword(user, password), client=self) def1.addCallbacks(callback=self.connected, errback=self.noLogin)
def noLogin(self, reason): reactor.callLater(5, self.connect) # retry after 5 seconds
def connected(self, perspective): self.presence = perspective perspective.notifyOnDisconnect(self.server_disconnected)
def server_disconnected(self, ref): reactor.callLater(5, self.connect) # retry after 5 seconds [...]
Luc
For what it's worth, I had a similar need and I don't pass objects around on PB (just data structures), so this class works out well for me. If the remote server isn't up or goes down, the returned deferred just waits. I tried to take out all the logic that relates to my application.
-Ken
class SimplePBProxy(object): """ A simple U{Perspective Brokerhttp://twistedmatrix.com/documents/current/howto/pb-intro} proxy for remote servers. This works similar to the xmlrpc proxy, but uses Perspective Broker.
Example::
hostname = 'remotebox' pb_port = 5053 proxy = SimplePBProxy(hostname, pb_port) deferred_object = proxy.callRemote('somecall')
B{NOTE:} Unlike the default Perspective Broker behavior, this doesn't wig out if the connection is lost. Deferred objects simple won't be returned until the remote server comes back up. Likewise, if the remote server isn't yet up, the deferred will simply be held open until the remote box is up. """ def __init__(self, host, port): """ @param host: Host to connect to. @rtype host: String
@param port: Port PB is on. @rtype port: Integer """ self.host = host self.port = port self.pending_calls = []
self.rootobj = None self.connect()
def connect(self): """ Used internally. Connects to remote server. """ def handle_error(failure): reactor.callLater(1, self.connect) factory = pb.PBClientFactory() factory.unsafeTracebacks = 1 reactor.connectTCP(self.host, self.port, factory) d = factory.getRootObject() d.addCallback(self._set_root_object) d.addErrback(handle_error) return d
def _set_root_object(self, rootobj): self.rootobj = rootobj
def pending_act(data, deferred): deferred.callback(data) def pending_err(failure, deferred): deferred.errback(failure)
for deferred, method, args, kwargs in self.pending_calls: d = self.callRemote(method, *args, **kwargs) d.addCallback(pending_act, deferred) d.addErrback(pending_err, deferred) self.pending_calls = []
def callRemote(self, method, *args, **kwargs): """ Call method on remote API. Method is a string. Any additional arguments and keyword arguments are passed to that method as arguments and keyword arguments. """ if not self.rootobj: d = Deferred() self.pending_calls.append((d, method, args, kwargs)) self.connect() return d
try: d = self.rootobj.callRemote(method, *args, **kwargs) d.addErrback(self._error_back, method, args, kwargs) return d except pb.DeadReferenceError: self.rootobj = None d = Deferred() self.pending_calls.append((d, method, args, kwargs)) self.connect() return d
def _error_back(self, failure, method, args, kwargs): if failure.type is twisted.spread.pb.PBConnectionLost: self.rootobj = None d = Deferred() self.pending_calls.append((d, method, args, kwargs)) self.connect() return d else: return failure
Dave Cook wrote:
I'm rendering the results of a remote method call:
def data_tableList(self, ctx, data): ... d = self.pbClientFactory.login(creds) d.addCallback(lambda object: object.callRemote("foo")) return d
Works, great if the PB server is up, but if it's down the browser just hangs. Is there a correct way to check that the connection is up at this point? This works:
if self.pbClientFactory._broker: d = self.pbClientFactory.login(creds) d.addCallback(lambda object: object.callRemote("getAlertReports")) else: d = [] return dbut seems like a hack. Also, ideally, I'd like to attempt a reconnection to the PB server at this point if it's not running. What's the best way to do that?
Thanks, Dave Cook
Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
participants (7)
-
Dave Cook -
Itamar Shtull-Trauring -
James Y Knight -
Jp Calderone -
Ken Kinder -
Luc Stepniewski -
Phil Mayers