Mailman 3 October 2012 - Twisted

Re: [Twisted-Python] Simpler Twisted deferred code via decorated callbacks
by Terry Jones Oct. 15, 2012

Oct. 15, 2012

Here's a description of what the 'errback' decorator in my code does. (BTW, I've just updated it.) Here's normal Twisted errback code, assuming you do from twisted.web.client import getPage and from twisted.python import log. def logGetPageError(url): def handleError(failure, url): log.msg('Could not fetch URL %s.' % url) return getPage(url).addErrback(handleError, url) Using the 'errback' decorator you could instead write: def logGetPageError(url): … [View More] @errback def handleError(failure, url): log.msg('Could not fetch URL %s.' % url) return handleError(getPage(url), url) As with the 'callback' decorator, the syntactic difference between the two in this case is minimal. The decorator returns a wrapping function that essentially does this: 0. Return a failed Deferred if the decorated function is called with no positional arguments (line 69). The reason for this is below. 1. Look at all its arguments. For any that are Deferreds, put them into a deferred list, which I'll call DL for this explanation. This is done for positional args (line 76) and keyword args (line 82). In our example, DL will contain the Deferred returned by getPage. 2. Suppose none of the arguments were Deferreds. If any of the args were instances of Failure (line 96), return a deferred that will fire with the result of calling the original wrapped (errback) function with all the passed arguments (at least one of which is a Failure). If no argument was a Failure, the original wrapped function is NOT called because there hasn't been any error. Instead (line 99), an already-fired Deferred (obtained via succeed) is returned with the value of the first positional argument. This is how the wrapping errback function passes the result along to its caller (if any). The other arguments to the wrapped function were obviously intended for that function, and are unused (this mirrors the way args passed to addErrback are unused if the errback is never run due to no error occurring). 3. If, as in our example, some of the arguments are deferreds (line 87) we have a DeferredList called DL that will eventually fire when all the Deferreds have fired (i.e., when all arguments are available for the wrapped function). Note that due to the use of addBoth (lines 79 and 84) non-Failure and Failure arguments coming from arguments that were Deferreds are both collected. These are collected, as they arrive, into a list (fargs) and a dictionary (fkw). Args that were not deferreds are already put into fargs and fkw (lines 78 and 86). 4. If DL fires with no errors (in our case, that means getPage returns a page), we act as we did in step 2: If any of the args are instances of Failure (line 89), the 'finish' callback added to DL (line 93) will return the result of calling the original wrapped (errback) function (handleError in our case) with all the passed arguments. If no argument was a Failure, the original wrapped function is NOT called because there hasn't been any error, and the callback on DL returns the first positional arg (on line 92). This last case corresponds to getPage returning successfully and the handleError errback not being called. The 'errback' decorator differs from the 'callback' decorator in that its wrapper function: - Collects all Failures in all arguments (including any coming from Deferreds, of course) in order to call the wrapped errback function. The 'callback' decorator OTOH immediately returns a failed Deferred as soon as any error happens (which is why its DeferredList uses fireOnOneErrback=True). - Must be called with at least one positional argument, so that in the case where nothing goes wrong with any Deferred argument (and no other arg is already a Failure) it has a value to pass back to its caller (if any). Both the wrapper functions sometimes do not call the wrapped function. This is symmetric and is as you'd expect: the 'callback' decorator doesn't call the wrapped (callback) function if there's any error, whereas the 'errback' decorator doesn't call the wrapped (errback) function unless there's an error. This is very similar to regular Twisted call/errback processing: if you're on the callback chain, errback functions aren't called & vice versa. Just as in regular Twisted code, a callback can produce a Failure or an errback can produce a non-Failure and you're back on the other chain. If you've read all this, great :-) It should be clear that you can roll up much more interesting cases, combining decorated callbacks and errbacks that accept multiple Deferred and non-Deferred arguments. Those are the cases where the syntactic / code complexity savings are much more apparent because (at least in many cases) there's no need to write code with addCallback, addErrback, or to use DeferredList. Some people's immediate reaction to all this is to tell me about inlineCallbacks. But this is very different, even though both share the universal decorator goal of making code look simpler / easier. inlineCallbacks lets you explicitly yield values from Deferreds. Unfortunately, anything can happen while you're context switched out before the yield yields. The decorators I've posted just wire up regular Twisted callback/errback processing chains, with no use of yield. There are other differences, but this post is already way long... Terry [View Less]

1 0

Re: [Twisted-Python] Simpler Twisted deferred code via decorated callbacks
by Terry Jones Oct. 15, 2012

Oct. 15, 2012

Here's a description of what the 'callback' decorator in my code does. Here's some normal Twisted code, assuming you do from twisted.web.client import getPage (yes, getPage is kind-of obsolete, but it's a concrete and conceptually simple deferred-returning function I like to use in examples). def pageLength(url): def _length(page): return len(page) return getPage(url).addCallback(_length) Using the 'callback' decorator you could instead write: def … [View More]pageLength(url): @callback def _length(page): return len(page) return _length(getPage(url) The syntactic difference in this case is minimal. The saving is more significant in other cases, but let's consider this simplest case. The decorator returns a wrapping function that essentially does this: 1. Look at all its arguments. For any that are Deferreds, put them into a deferred list, which I'll call DL for this explanation. This is done for positional args (line 25) and keyword args (line 31). 2. If no arguments are Deferreds, just call the original _length function with the original arguments, and return the result in a Deferred using maybeDeferred (line 44). So, any function that is called with a set of arguments that are not Deferred instances just returns a deferred that fires with the function result or fails if the function raises (maybeDeferred takes care of that for us). 3. The interesting part is when some of the arguments are deferreds (line 36). In that case, we have a DeferredList called DL that will eventually fire when all the arguments are available. The arguments are collected, as they arrive, into a list (fags) and a dictionary (fkw). Args that were not deferreds are already put into fargs and fkw (lines 30 and 35). 4. If DL fires normally, we can go ahead and call the wrapped original function with all the arguments it was supposed to receive. In our _length case, there was only one argument, a deferred returning the content of a web page, so the original _length gets called with the page content. A lambda to call the original function is added to DL as a callback (line 41), and DL is returned. Note that DL has already fired, so adding the callback function to call the original function results in the function being called immediately. By adding it as a callback to DL, we can return a deferred (DL) from the wrapper. 5. When DL is made, I pass fireOnOneErrback=True to it. So if any of the arguments to the wrapped function result in an error (i.e., in our example, getPage fails), then DL will fail immediately. In that case, an errback (getSubFailure) attached to DL will pull the original failure out of DL and return it. Because one of the arguments to the original function has failed to materialize, we obviously can't call the original function. By just returning DL (which has failed), the failure is propagated (in a deferred) to any caller we may have. If you look at the code for the 'callback' decorator in decorate.py, the above should help to make things clearer. I'm happy to answer any questions, of course. I hope the description makes it clear how the decorator works on a function called with multiple deferred arguments. It's quite like a DeferredList that you can put non-deferreds objects into and add a callback to (where the callback is the decorated function). I implemented that once years ago, but didn't bother to post it. The 'errback' decorator is similar in structure, but operates differently of course. I'll send a separate mail describing it. Terry [View Less]

1 0

Re: [Twisted-Python] Simpler Twisted deferred code via decorated callbacks
by Terry Jones Oct. 15, 2012

Oct. 15, 2012

Hi Naveen > This is a really nice approach. I'm glad you like it. I've learned over the years to avoid decorators, but I nevertheless think these ones are promising. > Are there any downsides or functionality that can't be accomplished using > this approach? The main thing that's a little different is that it doesn't (yet) help with situations where you might want finer-grained control over callbacks and errbacks, or to use addBoth. I didn't want to tackle everything at once. … [View More]

1 0

[Twisted-Python] Simpler Twisted deferred code via decorated callbacks
by Terry Jones Oct. 15, 2012

Oct. 15, 2012

This morning I was thinking about deferreds and how people find them difficult to grasp, but how they're conceptually simple once you get it. I guess most of us tell people a deferred is something to hold a result that hasn't arrived yet. Sometimes, though, deferreds do have a result in them immediately (e.g., using succeed or fail to get an already-fired deferred). I wondered if it might work to tell people to think of a deferred as really being the result. If that were literally true, … [View More]instead of writing: d = getPage(...) d.addErrback(errcheck, args) d.addCallback(cleanup, args) d.addCallback(reformat, args) return d We might write something like: result1 = getPage(...) result2 = errcheck(result1, args) result3 = cleanup(result2, args) return reformat(result3, args) And if you could write that, you could obviously instead write: return reformat(cleanup(errcheck(getPage(...), args), args), args) If we could write Twisted code that way, I think using deferreds would be simpler for people unfamiliar with them. In the style we're all used to, the programmer manually adds callbacks and errbacks. That's basically boilerplate. It gets worse when you then need to also use DeferredList, etc. It's a little confusing to read deferred code at first, because you need to know that the deferred result/failure is automatically passed as the first arg to callbacks/errbacks. It seems to take a year or more for people to finally realize how the callback & errback chains actually interact :-) Also, I wonder how comfortable programmers are with code ordered innermost function first, as in the normal d.addCallback(inner).addCallback(outer) Twisted style, versus outer(inner()), as in the line above. Anyway... I realized we CAN let people use the succinct style above, by putting boilerplate into decorators. I wrote two decorators, called (surprise!) callback and errback. You can do this: @errback def errcheck(failure, arg): ... @callback def cleanup(page, arg): ... @callback def reformat(page, arg): ... reformat(cleanup(errcheck(getPage(...), arg1), arg2), arg3) The deferred callback and errback chains are hooked up automatically. You still get a regular deferred back as the return value. And... the "deferred" aspect of the code (or at least the need to talk about or explain it) has conveniently vanished. You can also do things like func1(getDeferred1(), errcheck(func2(getDeferred2(), getDeferred3()))) This gets the result of deferreds 2 & 3 and (if neither fails) passes the result of calling func2 on both results through to func1, which is also called with the result of deferred 1. You don't need to use DeferredLists, as the decorator makes them for you. The errcheck function wont be called at all unless there's an error. That's nice compared to the verbose equivalent: d1 = DeferredList([getDeferred2(), getDeferred3()]) d1.addCallback(func2) d1.addErrback(errcheck) d2 = DeferredList([getDeferred1(), d1]) d2.addCallback(func1) Or the more compact but awkward: DeferredList([getDeferred(), DeferredList([getDeferred(), getDeferred()]).addCallback(func2).addErrback(errcheck)]).addCallback(func1) There's lots more that could be said about this, but that's enough for now. The code (surely not bulletproof) and some tests are at https://github.com/terrycojones/twisted-callback-decorators I'll add a README sometime soon. This is still pretty much proof of concept, and some it could be done slightly differently. I'm happy to discuss in more detail if people are interested. Terry [View Less]

2 1

[Twisted-Python] If you can read this, the mailing list is working again
by Itamar Turner-Trauring Oct. 15, 2012

Oct. 15, 2012

We've been having some delivery issues, if this arrives in your inbox then probably the mailing list is working again. -Itamar

1 0

[Twisted-Python] Weekly Bug Summary
by exarkun＠twistedmatrix.com Oct. 14, 2012

Oct. 14, 2012

1 0

[Twisted-Python] txtorcon 0.6
by meejah＠meejah.ca Oct. 12, 2012

Oct. 12, 2012

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 txtorcon 0.6 is now tagged and released. txtorcon is a Twisted-based Python asynchronous implementation of the Tor control protocol. It includes a state-tracking abstraction, configuration abstraction, Twisted endpoint support for hidden services, 96%+ unit-test coverage and many examples. New in this release: . debian packaging (mmaker); . psutil fully gone; . *changed API* for launch_tor() to use TorConfig instead of args; . TorConfig.… [View More]

1 0

[Twisted-Python] Weekly Bug Summary
by exarkun＠twistedmatrix.com Oct. 7, 2012

Oct. 7, 2012

1 0

[Twisted-Python] Consensus on how to handle "MySQL server has gone away"
by E S Oct. 5, 2012

Oct. 5, 2012

I have a service running that occasionally connects to a MySQL database. When there is no activity on it for some time, I eventually get the the error _mysql_exceptions.OperationalError: (2006, 'MySQL server has gone away') I have found a few posts on this group related to this issue http://twistedmatrix.com/pipermail/twisted-python/2007-July/015788.html http://twistedmatrix.com/pipermail/twisted-python/2007-October/016178.html as well as some tickets on the twisted homepage: http://… [View More]

7 9

Re: [Twisted-Python] Introducing Frack, a (partial) replacement for Trac
by Allen Short Oct. 4, 2012

Oct. 4, 2012

That's strange. I believe that's due to twistedmatrix.com using PB to connect different web server processes together, not anything directly related to the code in Frack. Try this URL? It's working for me currently. http://twistedmatrix.com/users/frack.twistd/ui/ticket.html?id=1956

1 0

[Twisted-Python] Introducing Frack, a (partial) replacement for Trac
by Allen Short Oct. 4, 2012

Oct. 4, 2012

Twisted's issue tracker is based on Trac, which has had increasingly bad performance over the years. Also, its code and deployment setup are so bad that everyone who has the ability to improve it has tried as hard as possible to avoid touching it at all. So I started writing a new issue tracker compatible with Trac's database, based on Twisted and more modern web technology. But it's incomplete and I'm not a JavaScript expert, so I need your help. If you've ever tried to contribute to Twisted … [View More]

1 0

Re: [Twisted-Python] dynamic FTP realm
by Tom Sheffler Oct. 1, 2012

Oct. 1, 2012

Hi Paul - I'm very new to twisted (I've only just started using it, so I apologise if > anything I ask is seemingly obvious!). > I'm looking to try and set up a FTP server which will serve a file > structure and files which don't exists on the server (the file structure > information is stored on cassandra, and the files are on s3). The file > structure and files will also be unique to the user, so different logins > will generate the file structure for that particular user, … [View More]

4 7