[Twisted-Python] Consensus on speed of deferred call/errback-style execution?
Last year someone mentioned to me that using defer.inlineCallbacks (versus writing a bunch of independent standalone callback functions) incurs some sort of (significant) speed penalty during execution. How accurate is that? More generally, there are many styles one can use to write and add callbacks to a deferred. Is there any consensus on the "best" way to do it, or are there rules of thumb that people have figured out, etc? I sometimes use defer.inlineCallbacks and then have a bunch of yields in a function. That saves writing and naming and addCallbacking a bunch of callback functions and can make for much nicer and easier to understand code. But the comment that using inlineCallbacks is slower sticks in my mind. Very briefly, you can add a bunch of callbacks to a deferred in multiple ways. E.g., 1. A bunch of top-level functions 2. A top-level function that nests callbacks that nest callbacks... 3. A bunch of top-level class methods 4. A top-level class method that nests callbacks... This is deliberately simplistic, and all the above can of course also be mixed. I wrote a little timing code to play with this (via timing a function that uses inlineCallbacks that yields a single passed deferred) but decided to ask here before going much further as there are lots of different avenues that could be explored. Terry
On Fri, 24 Oct 2008 17:25:29 +0200, Terry Jones <terry@jon.es> wrote:
Dunno.
A suite of benchmarks for Deferred has been on my to-do list for a long time. Aside from knowing which usage patterns are more efficient, this needs to be part of our testing process so we can evaluate optimization of Deferred and be aware of performance regressions we might introduce. I don't know if anyone else has done this already, but if you end up writing something, it'd be great if it were in a form which could be run automatically as part of our build process. Jean-Paul
On 03:25 pm, terry@jon.es wrote:
I don't believe there's been any extensive profiling of inlineCallbacks. At least, none that I'm aware of. I think what you are remembering is that inlineCallbacks generally results in an (often implicit, sometimes unintentional) loss of parallelism. For example, consider this idiomatic code: xd = foo() yd = foo() zd = foo() def myCallback(x, y, z): doStuff(x) + doStuff(y) / doStuff(z) gatherResults([x, y, z]).addCallback(myCallback) That does all three 'foo' operations in parallel, which is generally faster. But the idiomatic inlineCallbacks version: doStuff(yield foo()) + doStuff(yield foo()) / doStuff(yield foo()) while making all kinds of sexy use of expressions rather than statements, loses that parallelism: you don't get to the second foo() call until the first one has completed and its results have been processed. If each foo() call has a 500ms latency this can really add up. Of course you can work around this: xd = foo() yd = foo() zd = foo() doStuff(yield xd) + doStuff(yield yd) / doStuff(yield zd) but it can be difficult to remember to do so, and it starts looking a lot more like regular callback-style code. I think inlineCallbacks is neat, but its strength is really operations that are truly conversational; where the calling end of the conversation actually does need to wait for each step of an asynchronous conversation to complete before moving on to the next one. A dead giveaway that it's going to be awkward to make your inlineCallbacks appropriately parallel is if you start writing a 'for' loop that yields a Deferred in its body.
On Fri, 24 Oct 2008 17:25:29 +0200, Terry Jones <terry@jon.es> wrote:
Dunno.
A suite of benchmarks for Deferred has been on my to-do list for a long time. Aside from knowing which usage patterns are more efficient, this needs to be part of our testing process so we can evaluate optimization of Deferred and be aware of performance regressions we might introduce. I don't know if anyone else has done this already, but if you end up writing something, it'd be great if it were in a form which could be run automatically as part of our build process. Jean-Paul
On 03:25 pm, terry@jon.es wrote:
I don't believe there's been any extensive profiling of inlineCallbacks. At least, none that I'm aware of. I think what you are remembering is that inlineCallbacks generally results in an (often implicit, sometimes unintentional) loss of parallelism. For example, consider this idiomatic code: xd = foo() yd = foo() zd = foo() def myCallback(x, y, z): doStuff(x) + doStuff(y) / doStuff(z) gatherResults([x, y, z]).addCallback(myCallback) That does all three 'foo' operations in parallel, which is generally faster. But the idiomatic inlineCallbacks version: doStuff(yield foo()) + doStuff(yield foo()) / doStuff(yield foo()) while making all kinds of sexy use of expressions rather than statements, loses that parallelism: you don't get to the second foo() call until the first one has completed and its results have been processed. If each foo() call has a 500ms latency this can really add up. Of course you can work around this: xd = foo() yd = foo() zd = foo() doStuff(yield xd) + doStuff(yield yd) / doStuff(yield zd) but it can be difficult to remember to do so, and it starts looking a lot more like regular callback-style code. I think inlineCallbacks is neat, but its strength is really operations that are truly conversational; where the calling end of the conversation actually does need to wait for each step of an asynchronous conversation to complete before moving on to the next one. A dead giveaway that it's going to be awkward to make your inlineCallbacks appropriately parallel is if you start writing a 'for' loop that yields a Deferred in its body.
participants (3)
-
glyph＠divmod.com -
Jean-Paul Calderone -
Terry Jones