On Mar 23, 2011, at 9:24 AM, Kevin Horn wrote:

On Tue, Mar 22, 2011 at 9:03 PM, Glyph Lefkowitz <glyph@twistedmatrix.com> wrote: Why is Twisted's right hand blue? (Forget about being a beginner: I honestly don't even get this reference. Googling seems to suggest it has something to do with symptoms of heart disease and doesn't seem funny or relevant at all.)

My guess is that it has to do with Twister, the classic board(?) game.

Oh. Even worse! I don't want to get sued by Hasbro :).

Big wall of text incoming. If you're going to read any part of this email, search for *IMPORTANT* and read that part. Right now I'm stuck at creating a simple example for "deferred dependencies". On Tue, Mar 22, 2011 at 10:03 PM, Glyph Lefkowitz <glyph@twistedmatrix.com> wrote:

On Mar 21, 2011, at 9:30 PM, Jasper St. Pierre wrote:

...

On IRC, exarkun, glyph, spiv and idnar encouraged me to do a bit of work on the Defer documentation.

Yay!  This documentation could definitely use some work.

...

I kept get confused between the things like returning a Deferred from a callback and chainDeferred, which I found out wasn't that useful:

Yeah, chainDeferred is not a great method.  Now that Deferreds are non-recursive, I think it's purely worse than inserting an additional Deferred as a result from a callback.

My eventual goal is to reduce the number of documentation about defer down to a near-impossible two documents. I'm hoping to merge some of the good stuff of the other thousands of documents.

That would be absolutely great.

...

Thoughts so far?

While I applaud your intent, these drafts look quite rough.  The random interjections and asides in <http://magcius.mecheye.net/twisted/DeferHowTo-Fixup.html> seem distracting and confusing to me.  Trying to put myself in the mind of a newcomer, I find myself asking many questions which are irrelevant to what I'm trying to learn:

What's "async"?  Why is it hard?  (The original document mentions asynchronous stuff, but in the context of full english sentences.)

Should I mention blocking with something like urllib, then showcase another example library that uses regular callbacks?

Where are we going shopping?  What does shopping have to do with this?

Pop culture reference. Killed.

What's gevent?  Does this have something to do with Twisted? What's node.js?  This looks like Javascript, what does it have to do with Twisted, which I thought was in Python?

Placeholders.

(Now I've gotten distracted and I'm reading about gevent and node.js rather than making my Twisted application work and completing the Deferred tutorial.  Epic fail.  But, if I were to continue...)

What's "this pattern"?  Functions?  Don't lots of programs use functions?

I guess this is a bit obvious when you have first-class-functions as a language feature, but it's still a "pattern."

How do they use it? Why is it relevant?

I guess I'm stupid or slow, but it took a while for me to realize that Deferreds were basically a standardized callback mechanism. It's not really written anywhere on the tin: Deferred was to me a bit of an unobvious name for what it does, and before recently I've always associated it tightly to scheduling and the reactor.

Why is Twisted's right hand blue?  (Forget about being a beginner: I honestly don't even get this reference.  Googling seems to suggest it has something to do with symptoms of heart disease and doesn't seem funny or relevant at all.)

Another pop culture reference. Baleeted.

Why is the first explanation of what a Deferred is referred to as "technical mumbo-jumbo"? Is this really complicated?  If I am not super good at programming already, should I not be reading this?

It was prefixed with "The abstract" before. I put it back to "The abstract". To me, it seems like it's written in a way that makes sense only if you understand what a Deferred is, but it was useful, so I didn't rip it out. I see it as a paragraph that will make more sense as you're reading the article, and once you go back and understand it, there's that happy "snap" feeling as you get the concept.

What's an "operation"?  Does that mean 'function' or 'method' or some other special thing?  It says "most operations in Twisted return a Deferred"; but I've called lots of functions in Twisted which returned other objects before reading this tutorial, or returned None.  Were those actually Deferreds?

Yeah, I need to reword that. How about, "because Deferreds are a core part of Twisted, a lot of functions return them"? No... that's not good either.

Why do I "not know where this Deferred has been"?  Do Deferreds get dirty or broken somehow when I add multiple callbacks?  Should I avoid that?

Again, I fail at humor. Removed.

The Python examples in the current Deferred Reference are mostly runnable.  The ones that aren't, should be.  The documentation should stress that you can run these examples simply, and encourage the reader to download and experiment with them, and modify them to see what happens when they do things in a different order. Instead, the "fixup" changes the first example to rely on a fake library, which will raise exceptions if I try to run it, but doesn't actually explain that 'magiclib' isn't real.  This isn't a huge problem in and of itself (it is trying to demonstrate the "wrong" way to do things, after all) but it sets up the expectation that the rest of the examples are fake, too, and I shouldn't bother to run them.

Right now, it's a placeholder for that magic library that I haven't found yet.

I think the original document has plenty of issues, but these changes look like they've been written for people who already mostly understand Deferreds, but are having trouble catching some of the nuances, and need humor to diffuse their frustration and more examples to illustrate different usage patterns, rather than a fundamentally clearer or better explanation than what was offered before.

*IMPORTANT* Are there specific changes you find that could make it harder to read for newcomers? The concept of Deferreds isn't hard at all, once you understand what they are. The subtle nuances and bits of glue code that Twisted are the things that can trip someone up, and what I'm still learning. A small amount of very specific use cases for Deferreds happen in real-world code and I'd like to show the support that Twisted has for them built-in. My goals for this document are: 1) A list of guaranteed rules about Deferreds for reference at any time. 2) An introduction to those rules in a format that doesn't require knowledge of others. 3) Showing techniques or tricks that you can play by "exploiting" parts of those rules in the context of a contrived problem. 4) Showing the built-in support for it. This should help clear up my writing style a bit. I think in terms of separating abstraction layers; I always try separate a fact or rule from logic or a technique that can follow when you can exploit that fact (feel free to ask the people about 'evolution' in #python-offtopic). I also try to think of the code being very linear when it evolves: a new rule is added, you have a problem, you can exploit that rule with a specific technique, the technique is standardized. Example A: PROBLEM: You need to create a Deferred with a known result RULE: Callbacks will continue running after you've called "callback" or "errback" TECHNIQUE: You can create a Deferred, call 'callback' on it and return it, without any tricky business STANDARDIZED: twisted.internet.defer.success Example B: PROBLEM: You need to get the results from multiple Deferreds without blocking or too much linearity RULE: More than one Deferred can be created and 'run' at the same time TECHNIQUE: You can add a callback to a Deferred, take the result you get and save it in a list or dictionary STANDARDIZED: DeferredList, gatherResults The hardest part is creating simple, short, runnable code that introduces: a problem that doesn't seem silly, a rule that guides toward the solution, the technique that uses the rule to solve it. It was much easier when I could contrive examples of a network-enabled kitchen: recipes map pretty well to code, especially Twisted async: 1) Melt butter in a saucepan. When the butter is finished melting, put cocoa powder in. 2) Meanwhile, beat egg whites and sugar, and cream of tartar. 3) When both are done, put the chocolate mix in a Cuisinart, and fold in the egg whites. Here you have dependencies (butter needs to be melted before cocoa powder), multi-tasking (you don't want to wait for the butter while beating the eggs), and a way of knowing when things are done (so you can fold them into the cuisinart). ( Also, this is a real recipe, ableit simplified and it makes really easy, delicious chocolate mousse: http://articles.latimes.com/2008/feb/13/food/la-fo-watch13recafeb13 )

That makes sense, since based on what you've said on #twisted, that's basically the position you find yourself in :).  This document is supposed to be a tutorial though, explaining how to use Deferreds to users who really have no idea what they are (despite its unfortunate name, "Deferred Reference" - that should probably be changed). One thing I think is very good about this attempted rework, though, is the explanation of the motivation for having Deferreds at all, before explaining how they work.  In the current documentation, it's very unclear why we have such an object in the first place, or what the alternatives to it are.

Once I *knew* what a Deferred was, the other pieces started snapping into place.

However, the example presented makes it seem as though you really don't need Deferreds, because the only problem with the single-callback approach is handling errors. Another major motivation is the ability to return a Deferred through a system with several layers, changing the return value at each layer by post-processing it a bit.  (One possible example: a REST API that wants to deal with objects, and goes via a translation of [bytes from HTTP]->[JSON dicts/lists from parsing those bytes]->[domain-specific objects by converting JSON objects according to the particular API's spec].)

I never really thought about it before. I just realized right now, writing this email, that things like DeferredList aren't cleanly possible if the callback is tied to the request. Additionally, is the showcase of this in the fixup with "Multiple Callbacks" I did fine? The SQL to HTML example that was there before seemed a bit contrived, and I wanted to showcase it in a runnable snippet that required Twisted. We can't have them install a SQL server, so I used xml.minidom instead of lxml, even though know it's complete crap. When I'm done, I should replace the www.example.com URLs with files hosted on the Twisted doc site. Is there a way to point to generate a URL like that with Lore?

However, I think the need would be better illustrated with examples that can actually be run than with fake examples where we assume that the user knows how something like gevent works.  (Also: gevent doesn't actually work this way, for fetching web pages at least, so your example is wrong.  See <http://www.gevent.org/intro.html#monkey-patching>.)

I've replaced them with "Library A" and "Library B" placeholders. :)

It's pretty easy to write a fake implementation of 'fetchWebPageAsync' which squirrels away the callback somewhere that the example can call it later, and explain that with some handwaving where we say "and pretend that was actually some networking code fetching it".  For that matter, the reactor is introduced too early in the existing docs; we should demonstrate calling the callback synchronously, and then only later introduce a callLater.

I ripped out all the reactor code in the starting sections of the fixup on purpose, and showcased it with what I think is a clear example. When introducing the techniques afterwards, I'm going to gently ramp up the reactor code to be a bit more real-world.

Anyway I hope this wall of text did not discourage you - I just think you need some clearer goals for improving specific aspects of the documentation, and you should write those down first before trying to actually address them with more docs.

This email did exactly that. Thanks so much! P.S. I still have a *lot* to learn. Some tutoring on the subtleties of LoopingCall and coiterator/cooperator would be nice. I'm not going to even bother to try to explain how inlineCallbacks works to my brain right now. I've said enough erroneous fact in IRC. I'm going to need a lot more help in the future.

Thanks for your time,

I appreciate your time a lot more. You have things to do. I don't.

-glyph _______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

OK, I'm already making a lot of these changes, I should have a new version Up Soon (TM).

Placeholders.

Placeholders for what, though?

Placeholders for that magic lib, that I think I'm going to stop looking for and say, "demonstration purposes only" How do they use it? Why is it relevant? I guess I'm stupid or slow, but it took a while for me to realize that Deferreds were basically a standardized callback mechanism. It's not really written anywhere on the tin: Deferred was to me a bit of an unobvious name for what it does, and before recently I've always associated it tightly to scheduling and the reactor. No, this is a common problem. I think it would be great to address the definition a bit more comprehensively. Through example? It does seem to be a bit disconnected from the flow of the regular text.

Perhaps it would be better if it were laid out as a paragraph, and then each sentence, or clause, were examined more closely, perhaps with an accompanying code snippet to clarify it.

As I said, I don't want to explain that snippet. I just want to put it there, ignore it, and hopefully the user will understand more of it as the tutorial goes on.

Right now, it's a placeholder for that magic library that I haven't found yet.

I can see why you might want to do that. In the narrative flow between urllib and Deferreds, there is a missing step; you may need to fake that out. I'm just saying that you should be very clear and say "this is a fake example, merely for the purpose of illustration". I think that Tornado has some ugly callback-based stuff, but it would be better to leave this example fake than to try to teach users how to use that.

Will do.

Are there specific changes you find that could make it harder to read for newcomers?

I thought my previous message was a list of those :-(

Well, it was a draft, and I felt that most of my changes were already improvements over what was there. I didn't expect it to be "harder to read for newcomers".

The concept of Deferreds isn't hard at all, once you understand what they are. The subtle nuances and bits of glue code that Twisted are the things that can trip someone up, and what I'm still learning. A small amount of very specific use cases for Deferreds happen in real-world code and I'd like to show the support that Twisted has for them built-in.

My goals for this document are:

1) A list of guaranteed rules about Deferreds for reference at any time. 2) An introduction to those rules in a format that doesn't require knowledge of others. 3) Showing techniques or tricks that you can play by "exploiting" parts of those rules in the context of a contrived problem. 4) Showing the built-in support for it.

My first goal for this document would be a clear, concise explanation of what a Deferred is and why you need it.

As I said, when you understand what a Deferred is, it's not hard. The hard part is *how* to use it. The urllib, fake library, Deferred example is all I really want to go over about what a deferred is, because it's not hard. I want to: untie a lot of the code samples from the reactor, explain in terms of the two other examples that it's fundamentally the same thing, then and go on to the techniques like errbacks, chaining callbacks, and how separating the callback from the request allows things like DeferredList/gatherResults.

This should help clear up my writing style a bit. I think in terms of separating abstraction layers; I always try separate a fact or rule from logic or a technique that can follow when you can exploit that fact (feel free to ask the people about 'evolution' in #python-offtopic). I also try to think of the code being very linear when it evolves: a new rule is added, you have a problem, you can exploit that rule with a specific technique, the technique is standardized.

Example A: PROBLEM: You need to create a Deferred with a known result RULE: Callbacks will continue running after you've called "callback" or "errback" TECHNIQUE: You can create a Deferred, call 'callback' on it and return it, without any tricky business STANDARDIZED: twisted.internet.defer.success

Example B: PROBLEM: You need to get the results from multiple Deferreds without blocking or too much linearity RULE: More than one Deferred can be created and 'run' at the same time TECHNIQUE: You can add a callback to a Deferred, take the result you get and save it in a list or dictionary STANDARDIZED: DeferredList, gatherResults

These seem more like recipes to me than introductory documentation. Maybe they should be really close together, to try to drive the concept home, but it would be good to really get it clear in the reader's mind why they * fundamentally* need Deferreds, then to cover all the subtle *different* ways you might need them and how you could use them.

But you don't *fundamenally* *need *Deferreds at all. The new document isn't going to convince them that they're the right thing or wrong thing, just to show how to get along in their new home in Deferred-ville, USA. For the recipes, they're going to be reading real-world Twisted code, they better know what these recipes do. And if you can think of some subtly different ways to use them other than the generic "something is happening in the future, I better be prepared for it", I'll put it in the document and attempt to eat my hat.

*snip*

( Also, this is a real recipe, ableit simplified and it makes really easy, delicious chocolate mousse: http://articles.latimes.com/2008/feb/13/food/la-fo-watch13recafeb13 )

This looks like a fantastic example. It's comprehensible, concrete, not too long, and involves a strict metaphor for a real world situation, without mixing in any obscure technology. I would be happy if the entire Deferred tutorial were to be structured around it.

Except it really breaks that "runnable examples" thing. As I said before, recipes are a very convenient match for asynchronous code, but it's not realistic to try to map 1:1 to the other.

You could also tweak it to introduce additional concepts. For example, errbacks: "If the butter burns...".

If the butter burns.... what do I do? Turn off the heat, clean the pan, and restart some more butter? What if I run out of butter? Do I need a "goToStoreToFetchButter" function? Is that another tutorial I should write?

Once I *knew* what a Deferred was, the other pieces started snapping into place.

So it sounds like we're in agreement here: the existing document isn't clear enough about exactly what a Deferred is, it's described too formally and its uses aren't clear enough before we start diving into the technical specifics. Any modification should strive to make it super clear what it is and why you use it.

When I started with Twisted, I used Deferreds because "everything else in Twisted uses it". I didn't really *try* to understand it too much, either. Most of the time I used Twisted, I tried to hack it all up with inlineCallbacks and pretend it was urllib, and then complain in #twisted with a code snippet when it wasn't working. Use 'localhost' URLs and have the user run a 'twistd web' command line for

their server; that should be simple enough :).

For static pages, is there anything fundamentally wrong with " http://twistedmatrix.com/documents/11.1.0/static/pi.txt", if I can do it?

I'm really tired... the 'demo server' program is at http://p.mecheye.net/deferred-server.py/0 I coded it up in like 10 minutes. Definition of quality right there, yo. Hopefully no more noise tonight. On Mon, Mar 28, 2011 at 9:14 PM, Jasper St. Pierre <jstpierre@mecheye.net>wrote:

I'm tired. Link dump:

http://magcius.mecheye.net/twisted/DeferHowTo-Fixup.html http://magcius.mecheye.net/twisted/DeferHowTo-Fixup.lore

oldies are at

http://magcius.mecheye.net/twisted/DeferHowTo-Fixup-v1.html http://magcius.mecheye.net/twisted/DeferHowTo-Fixup-v1.lore

comments, etc.

Good catch, I really should mention that... On Tue, Mar 29, 2011 at 8:56 AM, Peter Hogg <dev@vokoda.com> wrote:

I haven't been following the thread, but just wanted to mention something that caught me out when I was learning to use deferreds.

Both the old deferred documentation and your fix-up fail to mention addCallbacks() until near the end, when in practice I find addCallbacks() to be much more useful than addCallback(). The diagram near the top demonstrating the deferred process also implies the use of addCallbacks(), even though prior to this only addCallback() and addErrback() have been mentioned. This confused me into thinking a consecutive addCallback().addErrback() would register a callback and an errback at the same point in the processing chain.

Apart from this point I found the old documentation very clear. If you really want the idiots guide I think you need animated diagrams, or something, but I don't see that happening.

Rgds, vokoda

Jasper St. Pierre wrote:

...

Glyph Lefkowitz

_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

On Tue, Mar 29, 2011 at 5:08 PM, Christopher Armstrong <radix@twistedmatrix.com> wrote:

I probably use addCallbacks once for every 30-100 addCallback or addErrback calls I do (counted by lines of code, not times they're executed). But that's just an off-the-cuff estimate.

For interest's sake, I just ran the numbers on my largest codebase (a proprietary application, sorry); ratio of addCallbacks to (addCallback + addErrback) is around 0.05. -- mithrandi, i Ainil en-Balandor, a faer Ambar

The problem that I have is that errback flow is awkward... the main difference is that addCallbacks will call the errback if its own callback fails, right? I can only see that really being useful by accident. On Wed, Mar 30, 2011 at 11:09 AM, Christopher Armstrong < radix@twistedmatrix.com> wrote:

On Wed, Mar 30, 2011 at 2:55 AM, Tristan Seligmann < mithrandi@mithrandi.net> wrote:

...

On Tue, Mar 29, 2011 at 5:08 PM, Christopher Armstrong <radix@twistedmatrix.com> wrote:

...

I probably once for every 30-'s 100 addCallback or addErrback calls I do (counted by lines of code, not times they're executed). But that's just an off-the-cuff estimate.

For interest's sake, I just ran the numbers on my largest codebase (a proprietary application, sorry); ratio of addCallbacks to (addCallback + addErrback) is around 0.05.

Funny, that's what I got on my proprietary application (0.0597) :-) It's a heavily inlineCallbacks-based codebase, and I expect there would be a much larger number of addCallback/addErrback calls if it weren't. Also, looking at the actual uses of addCallbacks (15), they were all written by people other than me (relative Twisted newbies) and I don't think I would have used it where it's used now. This, I think, indicates that we *should* focus more on addCallback and addErrback in the documentation, and stress that they are almost always what you want to use instead of addCallbacks, but definitely point out where addCallbacks is useful.

-- Christopher Armstrong http://radix.twistedmatrix.com/ http://planet-if.com/

_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

On Mar 30, 2011, at 11:09 AM, Christopher Armstrong wrote:

Funny, that's what I got on my proprietary application (0.0597) :-) It's a heavily inlineCallbacks-based codebase, and I expect there would be a much larger number of addCallback/addErrback calls if it weren't. Also, looking at the actual uses of addCallbacks (15), they were all written by people other than me (relative Twisted newbies) and I don't think I would have used it where it's used now. This, I think, indicates that we *should* focus more on addCallback and addErrback in the documentation, and stress that they are almost always what you want to use instead of addCallbacks, but definitely point out where addCallbacks is useful.

addCallbacks() is used in places where you'd need a try/except/else in synchronous code; much less frequently than you'd need a try/except, but still enough that it's important. However, in my experience, a novice's understanding of addCallbacks() is critically important to understanding other uses of Deferred as well; in particular, the "one chain of pairs of callback and errback" concept makes a lot of the behavior clear which might not otherwise be. If you just vaguely know about chains of callbacks, you can easily get confused. Some sample confusion that I believe I've heard over the years: thinking that addErrback only affects the previously-added callback, that there's only one errback for the whole chain but as many callbacks as you want, that the callback and errback chains are totally separate, and callbacks are run, then errbacks. Granted, these types of confusion require more than simply glossing over addCallbacks(), but I do think that emphasizing the pairs-of-callbacks structure helps people get a full understanding of what's going on more quickly. Of course, now that I've put this in a publicly-archived mailing list archive, a lack of understanding of addCallbacks won't be the problem any more. People will be confused, search for some terms related to this confusion, and read only that one preceding paragraph of this message, and say to themselves "oh, that's how Deferred works", somehow holding all of those wrong ideas in their head at once, forever. Hooray for the internet. -glyph

On Mar 30, 2011, at 2:51 PM, Tristan Seligmann wrote:

On Wed, Mar 30, 2011 at 7:20 PM, Glyph Lefkowitz <glyph@twistedmatrix.com> wrote:

...

However, in my experience, a novice's understanding of addCallbacks() is critically important to understanding other uses of Deferred as well; in particular, the "one chain of pairs of callback and errback" concept makes a lot of the behavior clear which might not otherwise be. If you just vaguely know about chains of callbacks, you can easily get confused.

Anecdotally, I didn't understand Deferred callback chains at all correctly until I discovered that addCallback and addErrback were implemented in terms of addCallbacks, and how they were implemented.

Great, so, my evidence-free claim and another your personal anecdotal experience agree; I believe this gives us what Science calls a "law", and thus the topic bears no further discussion :). Seriously though, this is pretty much exactly what I meant. You don't need to call addCallbacks a lot, but (A) sometimes you do and when you do need it it's important, and (B) it's important for people to understand early in the process even if they're not going to use it a lot.