Re: [Twisted-Python] Simpler Twisted deferred code via decorated callbacks

Here's a description of what the 'errback' decorator in my code does. (BTW, I've just updated it.) Here's normal Twisted errback code, assuming you do from twisted.web.client import getPage and from twisted.python import log. def logGetPageError(url): def handleError(failure, url): log.msg('Could not fetch URL %s.' % url) return getPage(url).addErrback(handleError, url) Using the 'errback' decorator you could instead write: def logGetPageError(url): @errback def handleError(failure, url): log.msg('Could not fetch URL %s.' % url) return handleError(getPage(url), url) As with the 'callback' decorator, the syntactic difference between the two in this case is minimal. The decorator returns a wrapping function that essentially does this: 0. Return a failed Deferred if the decorated function is called with no positional arguments (line 69). The reason for this is below. 1. Look at all its arguments. For any that are Deferreds, put them into a deferred list, which I'll call DL for this explanation. This is done for positional args (line 76) and keyword args (line 82). In our example, DL will contain the Deferred returned by getPage. 2. Suppose none of the arguments were Deferreds. If any of the args were instances of Failure (line 96), return a deferred that will fire with the result of calling the original wrapped (errback) function with all the passed arguments (at least one of which is a Failure). If no argument was a Failure, the original wrapped function is NOT called because there hasn't been any error. Instead (line 99), an already-fired Deferred (obtained via succeed) is returned with the value of the first positional argument. This is how the wrapping errback function passes the result along to its caller (if any). The other arguments to the wrapped function were obviously intended for that function, and are unused (this mirrors the way args passed to addErrback are unused if the errback is never run due to no error occurring). 3. If, as in our example, some of the arguments are deferreds (line 87) we have a DeferredList called DL that will eventually fire when all the Deferreds have fired (i.e., when all arguments are available for the wrapped function). Note that due to the use of addBoth (lines 79 and 84) non-Failure and Failure arguments coming from arguments that were Deferreds are both collected. These are collected, as they arrive, into a list (fargs) and a dictionary (fkw). Args that were not deferreds are already put into fargs and fkw (lines 78 and 86). 4. If DL fires with no errors (in our case, that means getPage returns a page), we act as we did in step 2: If any of the args are instances of Failure (line 89), the 'finish' callback added to DL (line 93) will return the result of calling the original wrapped (errback) function (handleError in our case) with all the passed arguments. If no argument was a Failure, the original wrapped function is NOT called because there hasn't been any error, and the callback on DL returns the first positional arg (on line 92). This last case corresponds to getPage returning successfully and the handleError errback not being called. The 'errback' decorator differs from the 'callback' decorator in that its wrapper function: - Collects all Failures in all arguments (including any coming from Deferreds, of course) in order to call the wrapped errback function. The 'callback' decorator OTOH immediately returns a failed Deferred as soon as any error happens (which is why its DeferredList uses fireOnOneErrback=True). - Must be called with at least one positional argument, so that in the case where nothing goes wrong with any Deferred argument (and no other arg is already a Failure) it has a value to pass back to its caller (if any). Both the wrapper functions sometimes do not call the wrapped function. This is symmetric and is as you'd expect: the 'callback' decorator doesn't call the wrapped (callback) function if there's any error, whereas the 'errback' decorator doesn't call the wrapped (errback) function unless there's an error. This is very similar to regular Twisted call/errback processing: if you're on the callback chain, errback functions aren't called & vice versa. Just as in regular Twisted code, a callback can produce a Failure or an errback can produce a non-Failure and you're back on the other chain. If you've read all this, great :-) It should be clear that you can roll up much more interesting cases, combining decorated callbacks and errbacks that accept multiple Deferred and non-Deferred arguments. Those are the cases where the syntactic / code complexity savings are much more apparent because (at least in many cases) there's no need to write code with addCallback, addErrback, or to use DeferredList. Some people's immediate reaction to all this is to tell me about inlineCallbacks. But this is very different, even though both share the universal decorator goal of making code look simpler / easier. inlineCallbacks lets you explicitly yield values from Deferreds. Unfortunately, anything can happen while you're context switched out before the yield yields. The decorators I've posted just wire up regular Twisted callback/errback processing chains, with no use of yield. There are other differences, but this post is already way long... Terry
participants (1)
-
Terry Jones