[Twisted-Python] ANN: deferred howto/tutorial
Hello all, I was sitting at Jury Duty today, trying to keep myself from the inevitable thoughts of suicide, and I started thinking about how to implement deferreds in C#. I started out by thinking about the logic one would need to use to implement deferreds in python. I wrote out in my notebook an implementation and when I got back home, I was suprised to discover that I wasn't terribly far off. I reflected upon this, and about what had confused me for so long about deferreds in general. ...and I was really stumped... ;) Deferreds are simple and elegant. After rewriting trial and dealing with all of its crazy exception handling, I realised that a lot of that code could have been made more succinct (if not more understandable) had it used deferreds to do the error handling and decision making. I think the thing that confuses most newbies is that deferreds to them seem like this BLACK BOX. It's this mysterious thing that is going to CALL YOUR CODE, and the rules about chaining and errback and callback chains are quite intimidating if you don't have a chunk of familiar looking code to transition you from python to Deferredy-Looking-Kool-Aid-Land. So what i did was write a tutorial on my deferred enlightenment...(pun intended?) I'd appreciate feedback, good or bad. You can find it in svn:HEAD, at doc/core/howto/tutorial/deferred_tutorial.xhtml Cheers! Jonathan D. Simms slyphon AT twistedmatrix DOT com
On Sat, 23 Oct 2004 00:08:21 -0600, Jonathan Simms <slyphon@twistedmatrix.com> wrote:
Therefore also available at http://svn.twistedmatrix.com/cvs/*checkout*/trunk/doc/core/howto/tutorial/deferred_tutorial.xhtml?rev=12147&root=Twisted via the Web interface, if you've not got a subversion client installed. Thanks, Jonathon - I've not sat & gone through it yet, but a quick glance suggested it's going to be useful!
On Sat, Oct 23, 2004, Jonathan Simms wrote:
That's not the place for it unfortunately, everything else in that directory is the finger tutorial. (ie it's not a general "tutorials" directory... but that's OK, the "howto" directory isn't really full of howtos either...) As best I can tell, there's no clear distinction between that and the "Using Deferreds" howto a directory up. I'll play around with merging them over the next hour or so and see if I'm right. -Mary
On Sat, Oct 23, 2004 at 08:29:20PM +0200, Mary Gardiner wrote:
Yeah, i'm sorry, Mary. I had a feeling that I was putting stuff in the wrong places when i committed it. I figured it was better to get it in, and have it moved around somewhere else than to wait, and possibly not get it included for 2.0. I really appreciate all the work you've done on documentation, and I didn't want you to get the impression that I was being careless with your realm of the source tree. Anyway, thanks! I'll leave the "exactly where" decision in your capable hands. -Jonathan
On Sun, Oct 24, 2004, Jonathan Simms wrote:
It's OK -- I'd rather have documentation in the wrong place than... no documentation!
Anyway, thanks! I'll leave the "exactly where" decision in your capable hands.
OK. I got lost in the async howto last night, so probably won't touch yours for a little while. Keep editing it where it is, and I'll move it when I've had a better look. -Mary
So what i did was write a tutorial on my deferred enlightenment...(pun intended?)
My way to grok deferreds was a detoured as well, when I started using twisted two months ago. First thing, I tried to avoid them. Until I reached a point where you wouldn't get to all that candy functionallity witouth using them. Then I found twisted.flow and thought this concept was much clearer to me. I used it and my code somehow worked. Until it didn't work anymore, and then I was screwed. Someone told me, I should rather use defgen, which I did. But when exeptions vanished in looping yields that yielded loops, I was lost again. At that point I rewrote the whole database backend using plain old deferreds. And then I understood them. Because they are simple and elegant, once you get the concept. What I missed in the docs, was a bit more concrete examples and styles of how to use deferreds in slightly more complex code. Most of the code is not class based, and you don't find branching or looping code that deals with deferreds. Both is not terribly difficult, but at first you really don't dare to know how a loop with deferreds will looks like. What I also would like to see is some coding practices of how to style deferred code. I got used to inner functions in class methods, which for me looks clearest, but I'm sure there are other practices. A short discussion on that in the docs would be great. I've attached an example of my style, maybe someone with a different style can rewrite the example and comment on pros and cons? --------------------------------------------------------------- class X: ... def y(self): # do some method initialisation ... # callbacks def oneBack(result): # extract someParams out of result ... return self.somethingDeferred().addCallback(twoBack, someParams) def twoBack(result, someParams): return "Foo" or raise Bar() return self.somethingDeferred().addCallback(oneBack) --------------------------------------------------------------- regards stefan.
On Sat, Oct 23, 2004 at 10:40:40PM +0200, stefan wrote: [...]
There's a cryptic note here that partly explains the usual style within Twisted itself: http://twistedmatrix.com/documents/howto/policy/coding-standard#auto13 Twisted style is usually: class X: def y(self): ... return self.somethingDeferred().addCallback(self._cbOneBack) def _cbOneBack(self, result): ... I believe the main reason for this is flexibility. Code in Twisted tends to be framework code or library code, rather than an actual application, so it is written with re-use in mind. You can't override a callback that's embedded within a method without overriding the whole method; making the callback a method on the class solves that. The "_cb" or "_eb" prefix signals that it's intended to be used as the callback or errback for something, and also that it's not an ordinary method that you would call directly. It also has other secondary advantages, like making setting breakpoints in pdb easier, although I can't say I've taken advantage of that very often (I typically set breakpoints by inserting "import pdb; pdb.set_trace()"), but I imagine some other developers have. A more important advantage (if you are strict on testing) is that you can unit test the behaviour of the callback more easily if you can call it directly. If nothing else, I like my functions to be as short as possible, and embedding a large callback within a method means I now have two large functions (the embedded function, and the method that contains it), rather than just one. The obvious disadvantage compared to your style is the loss of the direct visual association of which callbacks relate to which deferreds. I'd be moderately interested in hearing how other people cope with that, but I haven't really found it to be a problem. I expect that if I did, I'd just put comments like "# callback for deferred from self.frobnicate" at the top of each callback and errback. Large, hard-to-navigate classes with lots of methods can be a sign of a suboptimal design anyway. -Andrew.
On Sat, Oct 23, 2004, Andrew Bennetts wrote:
I find it's a problem when you start chaining deferreds by returning deferred from callbacks: not so much that it's difficult to find out where the callback is added, but that there's no visual representation that corresponds to the "nesting" of the callbacks. -Mary
On Sat, Oct 23, 2004 at 05:44:35PM -0400, Stephen Waterbury wrote:
That's how I think of it too. However, if you layout code like stefan does (placing callback definitions inside the other function definitions), then you will get visually nested callbacks. I suppose that's another argument against defining callbacks within functions ;) [Sidenote: in my usual mental model, "nesting" would be something you do with deferreds, rather than callbacks, i.e. it's what happens when a callback of one deferred returns some other deferred. And then there's "chaining" Deferreds with the "chainDeferred" method, which I always have to read the implementation of to figure out which way round I need to call it.] -Andrew.
stefan wrote:
I still prefer d = self.somethingDeferred() d.addCallback(oneBack) return d mostly because I tend to vomit when I see moshez write something like return self.somethingDeferred(foo, bar, baz, thud, quux).addCallback(oneBack, bar, baz, ).addErrback(twoBack, baz).addBoth(threeBack, foo) Also, I tend to use nested functions for small and mostly trivial things, like def firstOne((a,b)): return a and class-level functions for anything more complicated. Whenever it is not absolutely clear from the context that a function is a callback, I prefix the name with "cb", and if it's a class-level function, I start the name with an underscore to mark it an implementation detail. class X: ... def _cbOneBack(self, result): ... d = self.somethingDeferred() def format(result, someParams): return "Result is %r, %r" % (result, someParams) d.addCallback(format, someParams) return d def y(self): ... d = self.somethingDeferred() d.addCallback(self._cbOneBack) return d
On 24.10.2004, at 10:08, Tommi Virtanen wrote:
This is a nice visualization of the logical flow, much better than my first version.
Also, I tend to use nested functions for small and mostly trivial things
Make sense. A complex callback of course deserves it's own method. I just don't like cluttering up my class-namespace with three mostly trivial callbacks/errbacks per method. Thanks for everyone's feedback on the topic! And thanks for pointing out the "twisted-way" in the coding standards doc. stefan.
On Sat, 23 Oct 2004 00:08:21 -0600, Jonathan Simms <slyphon@twistedmatrix.com> wrote:
Therefore also available at http://svn.twistedmatrix.com/cvs/*checkout*/trunk/doc/core/howto/tutorial/deferred_tutorial.xhtml?rev=12147&root=Twisted via the Web interface, if you've not got a subversion client installed. Thanks, Jonathon - I've not sat & gone through it yet, but a quick glance suggested it's going to be useful!
On Sat, Oct 23, 2004, Jonathan Simms wrote:
That's not the place for it unfortunately, everything else in that directory is the finger tutorial. (ie it's not a general "tutorials" directory... but that's OK, the "howto" directory isn't really full of howtos either...) As best I can tell, there's no clear distinction between that and the "Using Deferreds" howto a directory up. I'll play around with merging them over the next hour or so and see if I'm right. -Mary
On Sat, Oct 23, 2004 at 08:29:20PM +0200, Mary Gardiner wrote:
Yeah, i'm sorry, Mary. I had a feeling that I was putting stuff in the wrong places when i committed it. I figured it was better to get it in, and have it moved around somewhere else than to wait, and possibly not get it included for 2.0. I really appreciate all the work you've done on documentation, and I didn't want you to get the impression that I was being careless with your realm of the source tree. Anyway, thanks! I'll leave the "exactly where" decision in your capable hands. -Jonathan
On Sun, Oct 24, 2004, Jonathan Simms wrote:
It's OK -- I'd rather have documentation in the wrong place than... no documentation!
Anyway, thanks! I'll leave the "exactly where" decision in your capable hands.
OK. I got lost in the async howto last night, so probably won't touch yours for a little while. Keep editing it where it is, and I'll move it when I've had a better look. -Mary
So what i did was write a tutorial on my deferred enlightenment...(pun intended?)
My way to grok deferreds was a detoured as well, when I started using twisted two months ago. First thing, I tried to avoid them. Until I reached a point where you wouldn't get to all that candy functionallity witouth using them. Then I found twisted.flow and thought this concept was much clearer to me. I used it and my code somehow worked. Until it didn't work anymore, and then I was screwed. Someone told me, I should rather use defgen, which I did. But when exeptions vanished in looping yields that yielded loops, I was lost again. At that point I rewrote the whole database backend using plain old deferreds. And then I understood them. Because they are simple and elegant, once you get the concept. What I missed in the docs, was a bit more concrete examples and styles of how to use deferreds in slightly more complex code. Most of the code is not class based, and you don't find branching or looping code that deals with deferreds. Both is not terribly difficult, but at first you really don't dare to know how a loop with deferreds will looks like. What I also would like to see is some coding practices of how to style deferred code. I got used to inner functions in class methods, which for me looks clearest, but I'm sure there are other practices. A short discussion on that in the docs would be great. I've attached an example of my style, maybe someone with a different style can rewrite the example and comment on pros and cons? --------------------------------------------------------------- class X: ... def y(self): # do some method initialisation ... # callbacks def oneBack(result): # extract someParams out of result ... return self.somethingDeferred().addCallback(twoBack, someParams) def twoBack(result, someParams): return "Foo" or raise Bar() return self.somethingDeferred().addCallback(oneBack) --------------------------------------------------------------- regards stefan.
On Sat, Oct 23, 2004 at 10:40:40PM +0200, stefan wrote: [...]
There's a cryptic note here that partly explains the usual style within Twisted itself: http://twistedmatrix.com/documents/howto/policy/coding-standard#auto13 Twisted style is usually: class X: def y(self): ... return self.somethingDeferred().addCallback(self._cbOneBack) def _cbOneBack(self, result): ... I believe the main reason for this is flexibility. Code in Twisted tends to be framework code or library code, rather than an actual application, so it is written with re-use in mind. You can't override a callback that's embedded within a method without overriding the whole method; making the callback a method on the class solves that. The "_cb" or "_eb" prefix signals that it's intended to be used as the callback or errback for something, and also that it's not an ordinary method that you would call directly. It also has other secondary advantages, like making setting breakpoints in pdb easier, although I can't say I've taken advantage of that very often (I typically set breakpoints by inserting "import pdb; pdb.set_trace()"), but I imagine some other developers have. A more important advantage (if you are strict on testing) is that you can unit test the behaviour of the callback more easily if you can call it directly. If nothing else, I like my functions to be as short as possible, and embedding a large callback within a method means I now have two large functions (the embedded function, and the method that contains it), rather than just one. The obvious disadvantage compared to your style is the loss of the direct visual association of which callbacks relate to which deferreds. I'd be moderately interested in hearing how other people cope with that, but I haven't really found it to be a problem. I expect that if I did, I'd just put comments like "# callback for deferred from self.frobnicate" at the top of each callback and errback. Large, hard-to-navigate classes with lots of methods can be a sign of a suboptimal design anyway. -Andrew.
On Sat, Oct 23, 2004, Andrew Bennetts wrote:
I find it's a problem when you start chaining deferreds by returning deferred from callbacks: not so much that it's difficult to find out where the callback is added, but that there's no visual representation that corresponds to the "nesting" of the callbacks. -Mary
On Sat, Oct 23, 2004 at 05:44:35PM -0400, Stephen Waterbury wrote:
That's how I think of it too. However, if you layout code like stefan does (placing callback definitions inside the other function definitions), then you will get visually nested callbacks. I suppose that's another argument against defining callbacks within functions ;) [Sidenote: in my usual mental model, "nesting" would be something you do with deferreds, rather than callbacks, i.e. it's what happens when a callback of one deferred returns some other deferred. And then there's "chaining" Deferreds with the "chainDeferred" method, which I always have to read the implementation of to figure out which way round I need to call it.] -Andrew.
stefan wrote:
I still prefer d = self.somethingDeferred() d.addCallback(oneBack) return d mostly because I tend to vomit when I see moshez write something like return self.somethingDeferred(foo, bar, baz, thud, quux).addCallback(oneBack, bar, baz, ).addErrback(twoBack, baz).addBoth(threeBack, foo) Also, I tend to use nested functions for small and mostly trivial things, like def firstOne((a,b)): return a and class-level functions for anything more complicated. Whenever it is not absolutely clear from the context that a function is a callback, I prefix the name with "cb", and if it's a class-level function, I start the name with an underscore to mark it an implementation detail. class X: ... def _cbOneBack(self, result): ... d = self.somethingDeferred() def format(result, someParams): return "Result is %r, %r" % (result, someParams) d.addCallback(format, someParams) return d def y(self): ... d = self.somethingDeferred() d.addCallback(self._cbOneBack) return d
On 24.10.2004, at 10:08, Tommi Virtanen wrote:
This is a nice visualization of the logical flow, much better than my first version.
Also, I tend to use nested functions for small and mostly trivial things
Make sense. A complex callback of course deserves it's own method. I just don't like cluttering up my class-namespace with three mostly trivial callbacks/errbacks per method. Thanks for everyone's feedback on the topic! And thanks for pointing out the "twisted-way" in the coding standards doc. stefan.
participants (7)
-
Andrew Bennetts -
Gwyn Evans -
Jonathan Simms -
Mary Gardiner -
stefan -
Stephen Waterbury -
Tommi Virtanen