[Twisted-Python] Dealing with an intermittent PB server
I'm rendering the results of a remote method call: def data_tableList(self, ctx, data): ... d = self.pbClientFactory.login(creds) d.addCallback(lambda object: object.callRemote("foo")) return d Works, great if the PB server is up, but if it's down the browser just hangs. Is there a correct way to check that the connection is up at this point? This works: if self.pbClientFactory._broker: d = self.pbClientFactory.login(creds) d.addCallback(lambda object: object.callRemote("getAlertReports")) else: d = [] return d but seems like a hack. Also, ideally, I'd like to attempt a reconnection to the PB server at this point if it's not running. What's the best way to do that? Thanks, Dave Cook
On Tue, 15 Feb 2005 19:56:15 -0800, Dave Cook <daverz@gmail.com> wrote:
Always always always use errbacks. *Especially* for Deferreds that can fail due to transient network conditions. def _cbLogin(self, object): return object.callRemote("foo") def _ebLogin(self, failure): return "Sorry, couldn't log in! (Try hitting ^R maybe)" def _ebCallFoo(self, failure): return "Sorry, calling foo failed! (Try hitting ^R maybe)" def data_tableList(self, ctx, data): ... d = self.pbClientFactory.login(creds) d.addCallbacks(self._cbLogin, self._ebLogin) d.addErrback(self._ebCallFoo) return d These errbacks are lame, but they cause the page to be renderable (and more generally, they let the asynchronous operation come to some kind of logical conclusion, instead of hanging forever, waiting for an error handler which will never arrive). Better ones could automatically retry the call, or inform an administrator of a possible misconfiguration, or some combination, or something even smarter.
clientConnectionFailed will be called on the ClientFactory, you should notice this and attempt to reconnect. Whether you fail any outstanding Deferreds which were waiting on the existing connection or let them survive to possibly succeed over the new one is up to you. Jp
On Tue, 15 Feb 2005 20:29:20 -0800 (PST), Jp Calderone <exarkun@divmod.com> wrote:
Always always always use errbacks. *Especially* for Deferreds that can fail due to transient network conditions.
Thanks. _ebLogin never seems to get called, though. I can see that tne PBClientFactory is stopped, so I assume that trying to login should trigger an error. Dave Cook
On Wed, 16 Feb 2005 11:56:20 -0500, James Y Knight <foom@fuhm.net> wrote:
Good point. If there's no way the Deferred can get back to application code from the framework, then the framework should take responsibility for error handling. This is the same as using a try/bare-except in framework code around synchronous calls into application code. Jp
On Wed, Feb 16, 2005 at 04:28:32AM +0000, Jp Calderone wrote:
See, that depends. I ran into this just today in fact, and found that if the server has gone away, then you'll get an exception *right then*, not via the errback: def do_stop(self): try: self.root.callRemote('stop', self).addCallbacks(self.stop, self.err) except pb.DeadReferenceError, re: self.stale() Why, why, why why why doesn't the DeadReferenceError go down the errback? As it is, the code has to be littered with both synchronous and asynchronous error handling, which is needlessly irritating, surely? The original poster may find his problem was a DeadReferenceError that's getting silently eaten (or just not GCed right away) by the framework, as I did. For the original poster, here's what I've got: class Thing: def __init__(self): self.root = None self.connect() def connect(self): f = pb.PBClientFactory() reactor.connectTCP(host, port, f) f.getRootObject().addCallbacks(self.connect_ok, self.connect_fail) def connect_ok(self, root): self.root = root def connect_fail(self, f): self.banner(f.getErrorMessage()) reactor.callLater(1, self.connect) def stale(self): self.root = None self.connect() def start(self): if not self.root: self.banner("Connection in progress, please wait") return try: self.root.callRemote('start').addCallbacks(self.start2, self.err) self.banner("sent...") except pb.DeadReferenceError, re: self.stale() self.banner("Connection dropped, reconnecting") def start2(self, dummy): self.banner("started") def err(self, f): self.banner(f.getErrorMessage())
On Thu, 17 Feb 2005 01:05:22 +0000, Phil Mayers <p.mayers@imperial.ac.uk> wrote:
Yes, this is unfortunate. Changing it will break programs that handle the error as it currently occurs, though. Brian Warner has been working on an improved version of PB for some time. It is mostly API compatible, and does a lot fewer irritating things of this sort.
Not quite. Twisted doesn't silently eat failures. But if you forget to add an errback, it might seem that way :) So this should just be taken as emphasis on my original point. If there _are_ framework pieces that are supposed to become responsible for a Deferred's error handling but aren't setting up errbacks, that's a bug. If you know of any, please report them on the bug tracker. (The Nevow example James Knight pointed out is probably one such case.) Jp
On Thu, 17 Feb 2005 01:05:22 +0000, Phil Mayers <p.mayers@imperial.ac.uk> wrote:
try: self.root.callRemote('start').addCallbacks(self.start2, self.err)
I never get this far. In my code: def data_tableList(self, ctx, data): ... d = self.pbClientFactory.login(creds).addCallbacks(self._cbLogin, self._ebLogin) return d Neither _cbLogin or _ebLogin ever get called, so there's no possiblity of the DeadReferenceError getting called. I guess this could be a Nevow bug (I hate bugging people if it's just my misunderstanding)? I think what I'll do is keep a reference to the original SSLClient object and check client.running before I try factory.login().
On Thu, Feb 17, 2005 at 11:31:24AM -0800, Dave Cook wrote:
That's odd. Even if Nevow discarded the deferred (not sure - I don't use it in data/render mode, my app is 100% livepage!) the callback/errback should still get called by the reactor. If I call "start" in my example with the server down, I get a Failure to errback with "errno 111 connection refused" (I see it in the LivePage, via my "alert" function) Is there any possibility there's a firewall between the Nevow middle and PB backend that's making TCP RST or ICMPs not work?
Wait, SSL? Hmm. Have you tried running over straight TCP just to see if there's a difference (I'm thinking error handling propagation etc. given the layered nature of SSL and comments seen on this list to the effect of the Twisted SSL implementation complexity)
On Thu, 17 Feb 2005 23:24:19 +0000, Phil Mayers <p.mayers@imperial.ac.uk> wrote:
Is there any possibility there's a firewall between the Nevow middle and PB backend that's making TCP RST or ICMPs not work?
When testing, I'm either running both server and client on the same machine or on machines on the same switch.
I'll try working up a test case, starting without SSL or authentication, and working my way up. Dave Cook
On Thu, 17 Feb 2005 11:31:24 -0800, Dave Cook <daverz@gmail.com> wrote:
Are you totally certain _cbLogin is never called? Remember, _ebLogin _will not_ be used as the errback for _cbLogin when it is added in the above manner: it will only handle errors from the .login() Deferred. To have _ebLogin handle all errors (except for those from itself): def data_tableList(self, ctx, data): ... d = self.pbClientFactory.login(creds) d.addCallback(self._cbLogin) d.addErrback(self._ebLogin) return d Or, to have _ebLogin continue handling errors from .login() and to have a separate error handler for _cbLogin and _ebLogin(): def data_tableList(self, ctx, data): ... d = self.pbClientFactory.login(creds) d.addCallbacks(self._cbLogin, self._ebLogin) d.addErrback(self._ebCallbacks) return d twisted.python.log.err is a really handy thing to use in place of self._ebCallbacks while debugging. It can very quickly show you an error you didn't even realize was happening (often a silly one - like declaring the wrong number of arguments to a callback function, causing a TypeError when Twisted tries to call it, making it appear as though the function is never called at all). Jp
On Thu, 17 Feb 2005 01:05:22 +0000, Phil Mayers <p.mayers@imperial.ac.uk> wrote:
Maybe I should do it the way you do here and not create the client factory until I need it rather than at the start of my application. How costly is it to create the client factories each time? Dave Cook
On Wednesday 16 February 2005 04:56, Dave Cook wrote:
Maybe it's because the connection succeeds, but *after* that, your server crashes or something else, so it's not catched by this code. You should use the notifyOnDisconnect() method for this. Here's a snippet of code I use regularly to do this: [...] def connect(self): factory = pb.PBClientFactory() reactor.connectTCP(distantServer, distantPort, factory) def1 = factory.login(credentials.UsernamePassword(user, password), client=self) def1.addCallbacks(callback=self.connected, errback=self.noLogin) def noLogin(self, reason): reactor.callLater(5, self.connect) # retry after 5 seconds def connected(self, perspective): self.presence = perspective perspective.notifyOnDisconnect(self.server_disconnected) def server_disconnected(self, ref): reactor.callLater(5, self.connect) # retry after 5 seconds [...] Luc -- Luc Stepniewski <luc.stepniewski@adelux.fr> Adelux - Securite, Linux Public key: <http://lstep.free.fr/pubkey.txt> Key BC0E3C2A fingerprint = A4FA466C68D27E46B427 07D083ED6340BC0E3C2A
For what it's worth, I had a similar need and I don't pass objects around on PB (just data structures), so this class works out well for me. If the remote server isn't up or goes down, the returned deferred just waits. I tried to take out all the logic that relates to my application. -Ken class SimplePBProxy(object): """ A simple U{Perspective Broker<http://twistedmatrix.com/documents/current/howto/pb-intro>} proxy for remote servers. This works similar to the xmlrpc proxy, but uses Perspective Broker. Example:: hostname = 'remotebox' pb_port = 5053 proxy = SimplePBProxy(hostname, pb_port) deferred_object = proxy.callRemote('somecall') B{NOTE:} Unlike the default Perspective Broker behavior, this doesn't wig out if the connection is lost. Deferred objects simple won't be returned until the remote server comes back up. Likewise, if the remote server isn't yet up, the deferred will simply be held open until the remote box is up. """ def __init__(self, host, port): """ @param host: Host to connect to. @rtype host: String @param port: Port PB is on. @rtype port: Integer """ self.host = host self.port = port self.pending_calls = [] self.rootobj = None self.connect() def connect(self): """ Used internally. Connects to remote server. """ def handle_error(failure): reactor.callLater(1, self.connect) factory = pb.PBClientFactory() factory.unsafeTracebacks = 1 reactor.connectTCP(self.host, self.port, factory) d = factory.getRootObject() d.addCallback(self._set_root_object) d.addErrback(handle_error) return d def _set_root_object(self, rootobj): self.rootobj = rootobj def pending_act(data, deferred): deferred.callback(data) def pending_err(failure, deferred): deferred.errback(failure) for deferred, method, args, kwargs in self.pending_calls: d = self.callRemote(method, *args, **kwargs) d.addCallback(pending_act, deferred) d.addErrback(pending_err, deferred) self.pending_calls = [] def callRemote(self, method, *args, **kwargs): """ Call method on remote API. Method is a string. Any additional arguments and keyword arguments are passed to that method as arguments and keyword arguments. """ if not self.rootobj: d = Deferred() self.pending_calls.append((d, method, args, kwargs)) self.connect() return d try: d = self.rootobj.callRemote(method, *args, **kwargs) d.addErrback(self._error_back, method, args, kwargs) return d except pb.DeadReferenceError: self.rootobj = None d = Deferred() self.pending_calls.append((d, method, args, kwargs)) self.connect() return d def _error_back(self, failure, method, args, kwargs): if failure.type is twisted.spread.pb.PBConnectionLost: self.rootobj = None d = Deferred() self.pending_calls.append((d, method, args, kwargs)) self.connect() return d else: return failure Dave Cook wrote:
On Tue, 15 Feb 2005 19:56:15 -0800, Dave Cook <daverz@gmail.com> wrote:
Always always always use errbacks. *Especially* for Deferreds that can fail due to transient network conditions. def _cbLogin(self, object): return object.callRemote("foo") def _ebLogin(self, failure): return "Sorry, couldn't log in! (Try hitting ^R maybe)" def _ebCallFoo(self, failure): return "Sorry, calling foo failed! (Try hitting ^R maybe)" def data_tableList(self, ctx, data): ... d = self.pbClientFactory.login(creds) d.addCallbacks(self._cbLogin, self._ebLogin) d.addErrback(self._ebCallFoo) return d These errbacks are lame, but they cause the page to be renderable (and more generally, they let the asynchronous operation come to some kind of logical conclusion, instead of hanging forever, waiting for an error handler which will never arrive). Better ones could automatically retry the call, or inform an administrator of a possible misconfiguration, or some combination, or something even smarter.
clientConnectionFailed will be called on the ClientFactory, you should notice this and attempt to reconnect. Whether you fail any outstanding Deferreds which were waiting on the existing connection or let them survive to possibly succeed over the new one is up to you. Jp
On Tue, 15 Feb 2005 20:29:20 -0800 (PST), Jp Calderone <exarkun@divmod.com> wrote:
Always always always use errbacks. *Especially* for Deferreds that can fail due to transient network conditions.
Thanks. _ebLogin never seems to get called, though. I can see that tne PBClientFactory is stopped, so I assume that trying to login should trigger an error. Dave Cook
On Wed, 16 Feb 2005 11:56:20 -0500, James Y Knight <foom@fuhm.net> wrote:
Good point. If there's no way the Deferred can get back to application code from the framework, then the framework should take responsibility for error handling. This is the same as using a try/bare-except in framework code around synchronous calls into application code. Jp
On Wed, Feb 16, 2005 at 04:28:32AM +0000, Jp Calderone wrote:
See, that depends. I ran into this just today in fact, and found that if the server has gone away, then you'll get an exception *right then*, not via the errback: def do_stop(self): try: self.root.callRemote('stop', self).addCallbacks(self.stop, self.err) except pb.DeadReferenceError, re: self.stale() Why, why, why why why doesn't the DeadReferenceError go down the errback? As it is, the code has to be littered with both synchronous and asynchronous error handling, which is needlessly irritating, surely? The original poster may find his problem was a DeadReferenceError that's getting silently eaten (or just not GCed right away) by the framework, as I did. For the original poster, here's what I've got: class Thing: def __init__(self): self.root = None self.connect() def connect(self): f = pb.PBClientFactory() reactor.connectTCP(host, port, f) f.getRootObject().addCallbacks(self.connect_ok, self.connect_fail) def connect_ok(self, root): self.root = root def connect_fail(self, f): self.banner(f.getErrorMessage()) reactor.callLater(1, self.connect) def stale(self): self.root = None self.connect() def start(self): if not self.root: self.banner("Connection in progress, please wait") return try: self.root.callRemote('start').addCallbacks(self.start2, self.err) self.banner("sent...") except pb.DeadReferenceError, re: self.stale() self.banner("Connection dropped, reconnecting") def start2(self, dummy): self.banner("started") def err(self, f): self.banner(f.getErrorMessage())
On Thu, 17 Feb 2005 01:05:22 +0000, Phil Mayers <p.mayers@imperial.ac.uk> wrote:
Yes, this is unfortunate. Changing it will break programs that handle the error as it currently occurs, though. Brian Warner has been working on an improved version of PB for some time. It is mostly API compatible, and does a lot fewer irritating things of this sort.
Not quite. Twisted doesn't silently eat failures. But if you forget to add an errback, it might seem that way :) So this should just be taken as emphasis on my original point. If there _are_ framework pieces that are supposed to become responsible for a Deferred's error handling but aren't setting up errbacks, that's a bug. If you know of any, please report them on the bug tracker. (The Nevow example James Knight pointed out is probably one such case.) Jp
On Thu, 17 Feb 2005 01:05:22 +0000, Phil Mayers <p.mayers@imperial.ac.uk> wrote:
try: self.root.callRemote('start').addCallbacks(self.start2, self.err)
I never get this far. In my code: def data_tableList(self, ctx, data): ... d = self.pbClientFactory.login(creds).addCallbacks(self._cbLogin, self._ebLogin) return d Neither _cbLogin or _ebLogin ever get called, so there's no possiblity of the DeadReferenceError getting called. I guess this could be a Nevow bug (I hate bugging people if it's just my misunderstanding)? I think what I'll do is keep a reference to the original SSLClient object and check client.running before I try factory.login().
On Thu, Feb 17, 2005 at 11:31:24AM -0800, Dave Cook wrote:
That's odd. Even if Nevow discarded the deferred (not sure - I don't use it in data/render mode, my app is 100% livepage!) the callback/errback should still get called by the reactor. If I call "start" in my example with the server down, I get a Failure to errback with "errno 111 connection refused" (I see it in the LivePage, via my "alert" function) Is there any possibility there's a firewall between the Nevow middle and PB backend that's making TCP RST or ICMPs not work?
Wait, SSL? Hmm. Have you tried running over straight TCP just to see if there's a difference (I'm thinking error handling propagation etc. given the layered nature of SSL and comments seen on this list to the effect of the Twisted SSL implementation complexity)
On Thu, 17 Feb 2005 23:24:19 +0000, Phil Mayers <p.mayers@imperial.ac.uk> wrote:
Is there any possibility there's a firewall between the Nevow middle and PB backend that's making TCP RST or ICMPs not work?
When testing, I'm either running both server and client on the same machine or on machines on the same switch.
I'll try working up a test case, starting without SSL or authentication, and working my way up. Dave Cook
On Thu, 17 Feb 2005 11:31:24 -0800, Dave Cook <daverz@gmail.com> wrote:
Are you totally certain _cbLogin is never called? Remember, _ebLogin _will not_ be used as the errback for _cbLogin when it is added in the above manner: it will only handle errors from the .login() Deferred. To have _ebLogin handle all errors (except for those from itself): def data_tableList(self, ctx, data): ... d = self.pbClientFactory.login(creds) d.addCallback(self._cbLogin) d.addErrback(self._ebLogin) return d Or, to have _ebLogin continue handling errors from .login() and to have a separate error handler for _cbLogin and _ebLogin(): def data_tableList(self, ctx, data): ... d = self.pbClientFactory.login(creds) d.addCallbacks(self._cbLogin, self._ebLogin) d.addErrback(self._ebCallbacks) return d twisted.python.log.err is a really handy thing to use in place of self._ebCallbacks while debugging. It can very quickly show you an error you didn't even realize was happening (often a silly one - like declaring the wrong number of arguments to a callback function, causing a TypeError when Twisted tries to call it, making it appear as though the function is never called at all). Jp
On Thu, 17 Feb 2005 01:05:22 +0000, Phil Mayers <p.mayers@imperial.ac.uk> wrote:
Maybe I should do it the way you do here and not create the client factory until I need it rather than at the start of my application. How costly is it to create the client factories each time? Dave Cook
On Wednesday 16 February 2005 04:56, Dave Cook wrote:
Maybe it's because the connection succeeds, but *after* that, your server crashes or something else, so it's not catched by this code. You should use the notifyOnDisconnect() method for this. Here's a snippet of code I use regularly to do this: [...] def connect(self): factory = pb.PBClientFactory() reactor.connectTCP(distantServer, distantPort, factory) def1 = factory.login(credentials.UsernamePassword(user, password), client=self) def1.addCallbacks(callback=self.connected, errback=self.noLogin) def noLogin(self, reason): reactor.callLater(5, self.connect) # retry after 5 seconds def connected(self, perspective): self.presence = perspective perspective.notifyOnDisconnect(self.server_disconnected) def server_disconnected(self, ref): reactor.callLater(5, self.connect) # retry after 5 seconds [...] Luc -- Luc Stepniewski <luc.stepniewski@adelux.fr> Adelux - Securite, Linux Public key: <http://lstep.free.fr/pubkey.txt> Key BC0E3C2A fingerprint = A4FA466C68D27E46B427 07D083ED6340BC0E3C2A
For what it's worth, I had a similar need and I don't pass objects around on PB (just data structures), so this class works out well for me. If the remote server isn't up or goes down, the returned deferred just waits. I tried to take out all the logic that relates to my application. -Ken class SimplePBProxy(object): """ A simple U{Perspective Broker<http://twistedmatrix.com/documents/current/howto/pb-intro>} proxy for remote servers. This works similar to the xmlrpc proxy, but uses Perspective Broker. Example:: hostname = 'remotebox' pb_port = 5053 proxy = SimplePBProxy(hostname, pb_port) deferred_object = proxy.callRemote('somecall') B{NOTE:} Unlike the default Perspective Broker behavior, this doesn't wig out if the connection is lost. Deferred objects simple won't be returned until the remote server comes back up. Likewise, if the remote server isn't yet up, the deferred will simply be held open until the remote box is up. """ def __init__(self, host, port): """ @param host: Host to connect to. @rtype host: String @param port: Port PB is on. @rtype port: Integer """ self.host = host self.port = port self.pending_calls = [] self.rootobj = None self.connect() def connect(self): """ Used internally. Connects to remote server. """ def handle_error(failure): reactor.callLater(1, self.connect) factory = pb.PBClientFactory() factory.unsafeTracebacks = 1 reactor.connectTCP(self.host, self.port, factory) d = factory.getRootObject() d.addCallback(self._set_root_object) d.addErrback(handle_error) return d def _set_root_object(self, rootobj): self.rootobj = rootobj def pending_act(data, deferred): deferred.callback(data) def pending_err(failure, deferred): deferred.errback(failure) for deferred, method, args, kwargs in self.pending_calls: d = self.callRemote(method, *args, **kwargs) d.addCallback(pending_act, deferred) d.addErrback(pending_err, deferred) self.pending_calls = [] def callRemote(self, method, *args, **kwargs): """ Call method on remote API. Method is a string. Any additional arguments and keyword arguments are passed to that method as arguments and keyword arguments. """ if not self.rootobj: d = Deferred() self.pending_calls.append((d, method, args, kwargs)) self.connect() return d try: d = self.rootobj.callRemote(method, *args, **kwargs) d.addErrback(self._error_back, method, args, kwargs) return d except pb.DeadReferenceError: self.rootobj = None d = Deferred() self.pending_calls.append((d, method, args, kwargs)) self.connect() return d def _error_back(self, failure, method, args, kwargs): if failure.type is twisted.spread.pb.PBConnectionLost: self.rootobj = None d = Deferred() self.pending_calls.append((d, method, args, kwargs)) self.connect() return d else: return failure Dave Cook wrote:
participants (7)
-
Dave Cook -
Itamar Shtull-Trauring -
James Y Knight -
Jp Calderone -
Ken Kinder -
Luc Stepniewski -
Phil Mayers