[Twisted-Python] Catching error in protocol.ReconnectingClientFactory
Hi there, I've been looking around for a solution for several hours now, and maybe I'm just missing something.. I've a factory that implements protocol.ReconnectingClientFactory which works just fine, however, whenever the program ends running, I get a bunch of Unhandled error in Deferred messages in my log for each disconnection that occurred during runtime, such as Failure: twisted.internet.error.ConnectionLost: ... Is there any way I could somehow "handle" these during runtime, so that my log wouldn't be spammed with this at the end.. (or avoid it being spammed somehow..) I really don't care what the solution is, I already have the following method that logs everything for me def clientConnectionLost(self, connector, reason): log.msg('Lost connection. Reason: %s' % reason) just annoying that it's still finds the need to print it out at the end. Anyway, I don't claim to be anywhere near proficient in Twisted, so any help would be greatly appreciated. Thanks a lot in advance. Martin
On Jun 11, 2012, at 8:01 AM, Martin wrote:
These messages are coming from some code which is issuing Deferreds, not from ReconnectingClientFactory. You need to add an errback to those Deferreds in your application code. There's no way to write a catch-all, just like there's no way to write a catch-all that can stop exceptions from being raised in basic Python. I can't say more without a code example, preferably a <http://sscce.org/>. -glyph
Hi, thanks for your reply. Indeed, the ReconnectingClientFactory is being sent the Failure object to def clientConnectionLost(self, connector, reason): log.msg('Lost connection. Reason: %s' % reason) protocol.ReconnectingClientFactory.clientConnectionLost(self, connector, reason) where "reason" is the Failure object that is "unhandled". is there anything I could do to it there to make it handled? Also, the documentation in http://twistedmatrix.com/documents/current/core/howto/defer.html says that one can do d.addErrback(err.log) to deal with unhandled errors, but what is d? In general, I'm not doing anything special.. I'm creating a ReconnectingClientFactory and connecting it to a server.. f = MyReconnectingClientFactory(..) reactor.connectSSL(url, port, f, ssl.ClientContextFactory()) reactor.run() Basically it. Any thoughts? Thanks a lot!
On 25/06/2012, at 10:16 PM, Martin <martin@webscio.net> wrote:
I believe glyph gave you the answer to that already. In your example above of d.addErrBack, d is the deferred object. Each of those needs to have an errback handler attached to it. If you don't want to see anything out of them, simply don't print anything.
On 25/06/12 22:27, Adrian Overbury wrote:
Well, kind of - I know that I need to do addErrBack on the deferred objects, I kind of knew that when I asked the original question having read the documentation on deferreds. What I still don't know though is how am I supposed to catch these deferreds - that is not being explained anywhere in the documentation I'm afraid.
On 26/06/2012, at 7:47 AM, Martin <martin@webscio.net> wrote:
The page and bookmark http://twistedmatrix.com/documents/current/core/howto/defer.html#auto4 explains it pretty well. When there's an error your errback function is called and a failure object is passed to it. You can handle it, drop it or pass it on as you please or seem appropriate. Hope that helps.
On Tue, Jun 26, 2012 at 5:47 AM, Martin <martin@webscio.net> wrote:
Hi Usually you don't need to catch a deferred object..adding some callbacks is enough.. I admit it might not be intuitive to understand defer concept at your first trial. But it is powerful & elegant. Regards gelin yan
On 06/25/2012 10:47 PM, Martin wrote:
What do you mean by "catch these deferreds"? A deferred is just a return value of a function, either in the Twisted API or your own code. For example: from twisted.web import client def ignore_err(failure): pass def work(): a_deferred = client.getPage(...) a_deferred.addErrback(ignore_err) If you are seeing unhandled errors in deferreds and you don't know which deferreds, you're probably throwing away a return value somewhere. The traceback should point you in the right directon.
On 25 Jun, 12:16 pm, martin@webscio.net wrote:
No suggestions as to how I could get rid of this? :(
I put together a minimal example of ReconnectingClientFactory for the connection refused case for you. It's attached. I don't see any errors logged. If your program differs from this, please modify the example until it produces the problem. Without this, it's very unlikely that anyone will be able to determine the problem or recommend a solution. For future questions, if you could put together this minimal example yourself, you'll probably find you get help more quickly. Here's the output I see when I run the program: 2012-06-26 10:07:26-0400 [-] Log opened. 2012-06-26 10:07:26-0400 [-] Starting factory <twisted.internet.protocol.ReconnectingClientFactory instance at 0x215ffc8> 2012-06-26 10:07:26-0400 [Uninitialized] <twisted.internet.ssl.Connector instance at 0x2164128> will retry in 2 seconds 2012-06-26 10:07:26-0400 [Uninitialized] Stopping factory <twisted.internet.protocol.ReconnectingClientFactory instance at 0x215ffc8> 2012-06-26 10:07:28-0400 [-] Starting factory <twisted.internet.protocol.ReconnectingClientFactory instance at 0x215ffc8> 2012-06-26 10:07:28-0400 [Uninitialized] <twisted.internet.ssl.Connector instance at 0x2164128> will retry in 4 seconds 2012-06-26 10:07:28-0400 [Uninitialized] Stopping factory <twisted.internet.protocol.ReconnectingClientFactory instance at 0x215ffc8> ^C2012-06-26 10:07:30-0400 [-] Received SIGINT, shutting down. 2012-06-26 10:07:30-0400 [-] Main loop terminated. Jean-Paul
On 26/06/12 15:12, exarkun@twistedmatrix.com wrote:
Thanks a lot for the suggestions! I tried plugging in a fake address (like you did) in my code and indeed I'm not getting the deferred message at the end. So apparently it only happens when a connection is actually established first and then lost. I'll try to figure out a minimal example where/why it starts happening (I'll probably know how to fix it then too.. I just thought there'd be an easier way).
On 28/06/12 12:46, Martin wrote:
And found it.. the ReconnectingClientFactory was using an instance of a LengthDelimitedStream as the protocol and that's where the deferred was coming from. Sorry for all the nagging, I should have figured it out myself.. but tbh getting the hang of Twisted is not so easy :) Cheers!
On Jun 11, 2012, at 8:01 AM, Martin wrote:
These messages are coming from some code which is issuing Deferreds, not from ReconnectingClientFactory. You need to add an errback to those Deferreds in your application code. There's no way to write a catch-all, just like there's no way to write a catch-all that can stop exceptions from being raised in basic Python. I can't say more without a code example, preferably a <http://sscce.org/>. -glyph
Hi, thanks for your reply. Indeed, the ReconnectingClientFactory is being sent the Failure object to def clientConnectionLost(self, connector, reason): log.msg('Lost connection. Reason: %s' % reason) protocol.ReconnectingClientFactory.clientConnectionLost(self, connector, reason) where "reason" is the Failure object that is "unhandled". is there anything I could do to it there to make it handled? Also, the documentation in http://twistedmatrix.com/documents/current/core/howto/defer.html says that one can do d.addErrback(err.log) to deal with unhandled errors, but what is d? In general, I'm not doing anything special.. I'm creating a ReconnectingClientFactory and connecting it to a server.. f = MyReconnectingClientFactory(..) reactor.connectSSL(url, port, f, ssl.ClientContextFactory()) reactor.run() Basically it. Any thoughts? Thanks a lot!
On 25/06/2012, at 10:16 PM, Martin <martin@webscio.net> wrote:
I believe glyph gave you the answer to that already. In your example above of d.addErrBack, d is the deferred object. Each of those needs to have an errback handler attached to it. If you don't want to see anything out of them, simply don't print anything.
On 25/06/12 22:27, Adrian Overbury wrote:
Well, kind of - I know that I need to do addErrBack on the deferred objects, I kind of knew that when I asked the original question having read the documentation on deferreds. What I still don't know though is how am I supposed to catch these deferreds - that is not being explained anywhere in the documentation I'm afraid.
On 26/06/2012, at 7:47 AM, Martin <martin@webscio.net> wrote:
The page and bookmark http://twistedmatrix.com/documents/current/core/howto/defer.html#auto4 explains it pretty well. When there's an error your errback function is called and a failure object is passed to it. You can handle it, drop it or pass it on as you please or seem appropriate. Hope that helps.
On Tue, Jun 26, 2012 at 5:47 AM, Martin <martin@webscio.net> wrote:
Hi Usually you don't need to catch a deferred object..adding some callbacks is enough.. I admit it might not be intuitive to understand defer concept at your first trial. But it is powerful & elegant. Regards gelin yan
On 06/25/2012 10:47 PM, Martin wrote:
What do you mean by "catch these deferreds"? A deferred is just a return value of a function, either in the Twisted API or your own code. For example: from twisted.web import client def ignore_err(failure): pass def work(): a_deferred = client.getPage(...) a_deferred.addErrback(ignore_err) If you are seeing unhandled errors in deferreds and you don't know which deferreds, you're probably throwing away a return value somewhere. The traceback should point you in the right directon.
On 25 Jun, 12:16 pm, martin@webscio.net wrote:
No suggestions as to how I could get rid of this? :(
I put together a minimal example of ReconnectingClientFactory for the connection refused case for you. It's attached. I don't see any errors logged. If your program differs from this, please modify the example until it produces the problem. Without this, it's very unlikely that anyone will be able to determine the problem or recommend a solution. For future questions, if you could put together this minimal example yourself, you'll probably find you get help more quickly. Here's the output I see when I run the program: 2012-06-26 10:07:26-0400 [-] Log opened. 2012-06-26 10:07:26-0400 [-] Starting factory <twisted.internet.protocol.ReconnectingClientFactory instance at 0x215ffc8> 2012-06-26 10:07:26-0400 [Uninitialized] <twisted.internet.ssl.Connector instance at 0x2164128> will retry in 2 seconds 2012-06-26 10:07:26-0400 [Uninitialized] Stopping factory <twisted.internet.protocol.ReconnectingClientFactory instance at 0x215ffc8> 2012-06-26 10:07:28-0400 [-] Starting factory <twisted.internet.protocol.ReconnectingClientFactory instance at 0x215ffc8> 2012-06-26 10:07:28-0400 [Uninitialized] <twisted.internet.ssl.Connector instance at 0x2164128> will retry in 4 seconds 2012-06-26 10:07:28-0400 [Uninitialized] Stopping factory <twisted.internet.protocol.ReconnectingClientFactory instance at 0x215ffc8> ^C2012-06-26 10:07:30-0400 [-] Received SIGINT, shutting down. 2012-06-26 10:07:30-0400 [-] Main loop terminated. Jean-Paul
On 26/06/12 15:12, exarkun@twistedmatrix.com wrote:
Thanks a lot for the suggestions! I tried plugging in a fake address (like you did) in my code and indeed I'm not getting the deferred message at the end. So apparently it only happens when a connection is actually established first and then lost. I'll try to figure out a minimal example where/why it starts happening (I'll probably know how to fix it then too.. I just thought there'd be an easier way).
On 28/06/12 12:46, Martin wrote:
And found it.. the ReconnectingClientFactory was using an instance of a LengthDelimitedStream as the protocol and that's where the deferred was coming from. Sorry for all the nagging, I should have figured it out myself.. but tbh getting the hang of Twisted is not so easy :) Cheers!
participants (6)
-
Adrian Overbury -
exarkun＠twistedmatrix.com -
gelin yan -
Glyph -
Martin -
Phil Mayers