[Twisted-Python] Passing additional arguments to errback
Dear list, I've found such a code example on "Stack Overflow" regarding errbacks: ___________________________ class YourExample(object): def your_example(self): self.agent = Agent(reactor, pool=pool) self.deferred = self.agent.request( 'GET', self.url, Headers({'User-Agent': ['Mozilla/5.0']}) ) self.deferred.addCallback(self.gotResponse).addErrback(self.gotBadResponse) def gotBadResponse(self,raised): """you might have cleanup code here, or mark the url as bad in the database, or something similar""" pass __________________________ Normally only Failure object is passed to gotBadResponse() . I would like to pass full response body to it - is it possible? Do I have to encapsulate the response body inside Failure object? Best regards Maciek
Cześć Maciek :) In general, you can pass extra arguments when you call addCallback(s) or addErrback. They will get passed to the callback. However, as a side note to that code example, do you understand the difference between .addCallbacks(cb, eb) and: .addCallback(cb).addErrback(eb) and: .addErrback(eb).addCallback(cb) ... Also, keep in mind that you only errback when there is an issue setting up the connection. If the server successfully responds with an error (say, a 404 Not Found, or something), the callback will be called with the response object. So, your question doesn't make a lot of sense to me: if the errback gets called, there's not really a response! Also, if you want to do scraping with Twisted, consider looking at Scrapy, a fully-featured web scraper that uses Twisted internally. pozdrawiam lvh
Laurens,
Cześć
Cześć! :) Thank you for your answer. I'm explicitly interested in the following combination:
.addCallback(cb).addErrback(eb)
If I understand correctly errback "eb" catches errors from both callback "cb", and from agent.request (agent.request errors pass through default empty errback that re-raises them).
Also, if you want to do scraping with Twisted, consider looking at Scrapy, a fully-featured web scraper that uses Twisted internally.
I'm working on CoAP protocol which runs on top of UDP. My actual code is request processing (but the problem is the same as in my previous post): _________________________________________________ def processRequest(self, request): (...) d = defer.succeed(request) d.addCallback(self.processBlock1) d.addCallback(self.dispatchRequest) def dispatchRequest(self, request): (...) resource = self.endpoint.getResourceFor(request) if resource is None: response = Message(code=NOT_FOUND, payload='Resource not found!' ) self.respond(response, request) return try: d = resource.render(request) except iot.error.UnallowedMethod: response = Message(code=METHOD_NOT_ALLOWED, payload='Method not allowed!') self.respond(response, request) return except iot.error.UnsupportedMethod: response = Message(code=NOT_IMPLEMENTED, payload='Method not implemented!') self.respond(response, request) return delayed_ack = reactor.callLater(EMPTY_ACK_DELAY, self.sendEmptyAck, request) d.addCallback(self.respond, request, delayed_ack) return d __________________________________________________ I would like to rewrite it to: __________________________________________________ def processRequest(self, request): (...) d = defer.succeed(request) d.addCallback(self.processBlock1) d.addCallback(self.dispatchRequest ).addErrback(self.handleRequestError) def dispatchRequest(self, request): (...) resource = self.endpoint.getResourceFor(request) if resource is None: raise NoResource() try: d = resource.render(request) except iot.error.UnallowedMethod: #Explicit re-rise for this example only raise UnallowedMethod() except iot.error.UnsupportedMethod: raise UnsupportedMethod() d.addCallback(self.respond, request) return d def handleRequestErrors(self, failure, request???) # handle exceptions, send error response to client _______________________________________________ I would like to handle Exceptions in handleRequestErrors(). However handleRequestErrors() has to send a response to the client, so it needs the request, and I don't know how to pass it from inside dispatchRequest(). I see two possibilities: 1. Pass request inside Failure object 2. Leave the original code (process errors inside callback) Which solution is more elegant? Best Regards Maciek
Hi, On Thu, Sep 5, 2013 at 9:24 AM, Maciej Wasilak <wasilak@gmail.com> wrote:
Yes, that's exactly what it does :) The difference being that with .addCallbacks(cb, eb), errors in cb would not be caught by eb.
You don't have to do it from in there. You can do .addErrback(handleErrors, request), since it's all the same request object, right?
Both of those would work, but see above :) Two random pieces of code review: 1. It seems self.endpoint is a t.w.s.Site object. That's kind of confusing, since twisted has an "endpoint" concept that's one step removed from a Site (endpoints connect or listen with factories, a Site is a factory). You might want to reconsider that name :) 2. It's kind of strange to start with defer.succeed() and then start making a callback chain IMHO, but it's not wrong, really. cheers lvh
Laurens,
Aaargh! I see the problem now. I wrote everything as part of the Protocol class (D&D - Deferreds&Dictionaries), when I should have extracted request functionality into separate class. Then I can save request body as a class member between callback and errback. Thanks!!!
Endpoint is the official name in draft: http://tools.ietf.org/html/draft-ietf-core-coap-18 I'll rename it to coap_endpoint to avoid confusion
2. It's kind of strange to start with defer.succeed() and then start making a callback chain IMHO, but it's not wrong, really.
I've recently understood callback chaining and I really like the idea. I guess it's this old proverb about having a hammer, and seeing nails everywhere. I'll try to come up with something better :) . Case closed - thank you very much! Regards Maciek
On Thu, Sep 5, 2013 at 12:00 PM, Maciej Wasilak <wasilak@gmail.com> wrote:
Well, yes, you should, but still keep in mind that you can actually just pass the request like so: .addErrback(handleErrors, request) You don't really have to save the request itself anywhere as an attribute.
No problem.
Case closed - thank you very much!
Glad to have helped.
Regards Maciek
cheers lvh
Hello, just one more question: Well, yes, you should, but still keep in mind that you can actually just
if I understand correctly, when I put this instruction into my callback, then the errback will catch errors only from the callback, and not from agent.request, right? Such code seems hard to debug. Is this the standard procedure in Twisted to add callbacks/errbacks inside other callbacks? Best Regards Maciek
Cześć Maciek :) In general, you can pass extra arguments when you call addCallback(s) or addErrback. They will get passed to the callback. However, as a side note to that code example, do you understand the difference between .addCallbacks(cb, eb) and: .addCallback(cb).addErrback(eb) and: .addErrback(eb).addCallback(cb) ... Also, keep in mind that you only errback when there is an issue setting up the connection. If the server successfully responds with an error (say, a 404 Not Found, or something), the callback will be called with the response object. So, your question doesn't make a lot of sense to me: if the errback gets called, there's not really a response! Also, if you want to do scraping with Twisted, consider looking at Scrapy, a fully-featured web scraper that uses Twisted internally. pozdrawiam lvh
Laurens,
Cześć
Cześć! :) Thank you for your answer. I'm explicitly interested in the following combination:
.addCallback(cb).addErrback(eb)
If I understand correctly errback "eb" catches errors from both callback "cb", and from agent.request (agent.request errors pass through default empty errback that re-raises them).
Also, if you want to do scraping with Twisted, consider looking at Scrapy, a fully-featured web scraper that uses Twisted internally.
I'm working on CoAP protocol which runs on top of UDP. My actual code is request processing (but the problem is the same as in my previous post): _________________________________________________ def processRequest(self, request): (...) d = defer.succeed(request) d.addCallback(self.processBlock1) d.addCallback(self.dispatchRequest) def dispatchRequest(self, request): (...) resource = self.endpoint.getResourceFor(request) if resource is None: response = Message(code=NOT_FOUND, payload='Resource not found!' ) self.respond(response, request) return try: d = resource.render(request) except iot.error.UnallowedMethod: response = Message(code=METHOD_NOT_ALLOWED, payload='Method not allowed!') self.respond(response, request) return except iot.error.UnsupportedMethod: response = Message(code=NOT_IMPLEMENTED, payload='Method not implemented!') self.respond(response, request) return delayed_ack = reactor.callLater(EMPTY_ACK_DELAY, self.sendEmptyAck, request) d.addCallback(self.respond, request, delayed_ack) return d __________________________________________________ I would like to rewrite it to: __________________________________________________ def processRequest(self, request): (...) d = defer.succeed(request) d.addCallback(self.processBlock1) d.addCallback(self.dispatchRequest ).addErrback(self.handleRequestError) def dispatchRequest(self, request): (...) resource = self.endpoint.getResourceFor(request) if resource is None: raise NoResource() try: d = resource.render(request) except iot.error.UnallowedMethod: #Explicit re-rise for this example only raise UnallowedMethod() except iot.error.UnsupportedMethod: raise UnsupportedMethod() d.addCallback(self.respond, request) return d def handleRequestErrors(self, failure, request???) # handle exceptions, send error response to client _______________________________________________ I would like to handle Exceptions in handleRequestErrors(). However handleRequestErrors() has to send a response to the client, so it needs the request, and I don't know how to pass it from inside dispatchRequest(). I see two possibilities: 1. Pass request inside Failure object 2. Leave the original code (process errors inside callback) Which solution is more elegant? Best Regards Maciek
Hi, On Thu, Sep 5, 2013 at 9:24 AM, Maciej Wasilak <wasilak@gmail.com> wrote:
Yes, that's exactly what it does :) The difference being that with .addCallbacks(cb, eb), errors in cb would not be caught by eb.
You don't have to do it from in there. You can do .addErrback(handleErrors, request), since it's all the same request object, right?
Both of those would work, but see above :) Two random pieces of code review: 1. It seems self.endpoint is a t.w.s.Site object. That's kind of confusing, since twisted has an "endpoint" concept that's one step removed from a Site (endpoints connect or listen with factories, a Site is a factory). You might want to reconsider that name :) 2. It's kind of strange to start with defer.succeed() and then start making a callback chain IMHO, but it's not wrong, really. cheers lvh
Laurens,
Aaargh! I see the problem now. I wrote everything as part of the Protocol class (D&D - Deferreds&Dictionaries), when I should have extracted request functionality into separate class. Then I can save request body as a class member between callback and errback. Thanks!!!
Endpoint is the official name in draft: http://tools.ietf.org/html/draft-ietf-core-coap-18 I'll rename it to coap_endpoint to avoid confusion
2. It's kind of strange to start with defer.succeed() and then start making a callback chain IMHO, but it's not wrong, really.
I've recently understood callback chaining and I really like the idea. I guess it's this old proverb about having a hammer, and seeing nails everywhere. I'll try to come up with something better :) . Case closed - thank you very much! Regards Maciek
On Thu, Sep 5, 2013 at 12:00 PM, Maciej Wasilak <wasilak@gmail.com> wrote:
Well, yes, you should, but still keep in mind that you can actually just pass the request like so: .addErrback(handleErrors, request) You don't really have to save the request itself anywhere as an attribute.
No problem.
Case closed - thank you very much!
Glad to have helped.
Regards Maciek
cheers lvh
Hello, just one more question: Well, yes, you should, but still keep in mind that you can actually just
if I understand correctly, when I put this instruction into my callback, then the errback will catch errors only from the callback, and not from agent.request, right? Such code seems hard to debug. Is this the standard procedure in Twisted to add callbacks/errbacks inside other callbacks? Best Regards Maciek
participants (3)
-
Laurens Van Houtven -
Maciej Wasilak -
Phil Mayers