Hi Terry, This is a really nice approach. Thanks for sharing! Are there any downsides or functionality that can't be accomplished using this approach? This combined with the generator approach to deferreds will make it easier to reason about the code flow. Naveen On Sun, Oct 14, 2012 at 4:40 PM, Terry Jones <terry@jon.es> wrote:

This morning I was thinking about deferreds and how people find them difficult to grasp, but how they're conceptually simple once you get it. I guess most of us tell people a deferred is something to hold a result that hasn't arrived yet. Sometimes, though, deferreds do have a result in them immediately (e.g., using succeed or fail to get an already-fired deferred).

I wondered if it might work to tell people to think of a deferred as really being the result. If that were literally true, instead of writing:

d = getPage(...) d.addErrback(errcheck, args) d.addCallback(cleanup, args) d.addCallback(reformat, args) return d

We might write something like:

result1 = getPage(...) result2 = errcheck(result1, args) result3 = cleanup(result2, args) return reformat(result3, args)

And if you could write that, you could obviously instead write:

return reformat(cleanup(errcheck(getPage(...), args), args), args)

If we could write Twisted code that way, I think using deferreds would be simpler for people unfamiliar with them.

In the style we're all used to, the programmer manually adds callbacks and errbacks. That's basically boilerplate. It gets worse when you then need to also use DeferredList, etc. It's a little confusing to read deferred code at first, because you need to know that the deferred result/failure is automatically passed as the first arg to callbacks/errbacks. It seems to take a year or more for people to finally realize how the callback & errback chains actually interact :-) Also, I wonder how comfortable programmers are with code ordered innermost function first, as in the normal d.addCallback(inner).addCallback(outer) Twisted style, versus outer(inner()), as in the line above.

Anyway... I realized we CAN let people use the succinct style above, by putting boilerplate into decorators. I wrote two decorators, called (surprise!) callback and errback. You can do this:

@errback def errcheck(failure, arg): ...

@callback def cleanup(page, arg): ...

@callback def reformat(page, arg): ...

reformat(cleanup(errcheck(getPage(...), arg1), arg2), arg3)

The deferred callback and errback chains are hooked up automatically. You still get a regular deferred back as the return value.

And... the "deferred" aspect of the code (or at least the need to talk about or explain it) has conveniently vanished.

You can also do things like

func1(getDeferred1(), errcheck(func2(getDeferred2(), getDeferred3())))

This gets the result of deferreds 2 & 3 and (if neither fails) passes the result of calling func2 on both results through to func1, which is also called with the result of deferred 1. You don't need to use DeferredLists, as the decorator makes them for you. The errcheck function wont be called at all unless there's an error.

That's nice compared to the verbose equivalent:

d1 = DeferredList([getDeferred2(), getDeferred3()]) d1.addCallback(func2) d1.addErrback(errcheck) d2 = DeferredList([getDeferred1(), d1]) d2.addCallback(func1)

Or the more compact but awkward:

DeferredList([getDeferred(), DeferredList([getDeferred(), getDeferred()]).addCallback(func2).addErrback(errcheck)]).addCallback(func1)

There's lots more that could be said about this, but that's enough for now. The code (surely not bulletproof) and some tests are at https://github.com/terrycojones/twisted-callback-decorators

I'll add a README sometime soon. This is still pretty much proof of concept, and some it could be done slightly differently. I'm happy to discuss in more detail if people are interested.

Terry

_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Hi Terry, This is a really nice approach. Thanks for sharing! Are there any downsides or functionality that can't be accomplished using this approach? This combined with the generator approach to deferreds will make it easier to reason about the code flow. Naveen On Sun, Oct 14, 2012 at 4:40 PM, Terry Jones <terry@jon.es> wrote:

This morning I was thinking about deferreds and how people find them difficult to grasp, but how they're conceptually simple once you get it. I guess most of us tell people a deferred is something to hold a result that hasn't arrived yet. Sometimes, though, deferreds do have a result in them immediately (e.g., using succeed or fail to get an already-fired deferred).

I wondered if it might work to tell people to think of a deferred as really being the result. If that were literally true, instead of writing:

d = getPage(...) d.addErrback(errcheck, args) d.addCallback(cleanup, args) d.addCallback(reformat, args) return d

We might write something like:

result1 = getPage(...) result2 = errcheck(result1, args) result3 = cleanup(result2, args) return reformat(result3, args)

And if you could write that, you could obviously instead write:

return reformat(cleanup(errcheck(getPage(...), args), args), args)

If we could write Twisted code that way, I think using deferreds would be simpler for people unfamiliar with them.

In the style we're all used to, the programmer manually adds callbacks and errbacks. That's basically boilerplate. It gets worse when you then need to also use DeferredList, etc. It's a little confusing to read deferred code at first, because you need to know that the deferred result/failure is automatically passed as the first arg to callbacks/errbacks. It seems to take a year or more for people to finally realize how the callback & errback chains actually interact :-) Also, I wonder how comfortable programmers are with code ordered innermost function first, as in the normal d.addCallback(inner).addCallback(outer) Twisted style, versus outer(inner()), as in the line above.

Anyway... I realized we CAN let people use the succinct style above, by putting boilerplate into decorators. I wrote two decorators, called (surprise!) callback and errback. You can do this:

@errback def errcheck(failure, arg): ...

@callback def cleanup(page, arg): ...

@callback def reformat(page, arg): ...

reformat(cleanup(errcheck(getPage(...), arg1), arg2), arg3)

The deferred callback and errback chains are hooked up automatically. You still get a regular deferred back as the return value.

And... the "deferred" aspect of the code (or at least the need to talk about or explain it) has conveniently vanished.

You can also do things like

func1(getDeferred1(), errcheck(func2(getDeferred2(), getDeferred3())))

This gets the result of deferreds 2 & 3 and (if neither fails) passes the result of calling func2 on both results through to func1, which is also called with the result of deferred 1. You don't need to use DeferredLists, as the decorator makes them for you. The errcheck function wont be called at all unless there's an error.

That's nice compared to the verbose equivalent:

d1 = DeferredList([getDeferred2(), getDeferred3()]) d1.addCallback(func2) d1.addErrback(errcheck) d2 = DeferredList([getDeferred1(), d1]) d2.addCallback(func1)

Or the more compact but awkward:

DeferredList([getDeferred(), DeferredList([getDeferred(), getDeferred()]).addCallback(func2).addErrback(errcheck)]).addCallback(func1)

There's lots more that could be said about this, but that's enough for now. The code (surely not bulletproof) and some tests are at https://github.com/terrycojones/twisted-callback-decorators

I'll add a README sometime soon. This is still pretty much proof of concept, and some it could be done slightly differently. I'm happy to discuss in more detail if people are interested.

Terry

_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python