[Twisted-Python] Questions about adding documentation
![](https://secure.gravatar.com/avatar/a7267ede9d6ea0117c5438c398a1a212.jpg?s=120&d=mm&r=g)
Hi everyone, I would like to help Twisted by adding documentation or reviewing existing documentation. However, I don't see many (any?) unassigned tickets regarding documentation of specific items, and because I am quite new to Twisted this makes it hard for me to determine where you wish I would focus my attention. Where should I focus my attention? Want to open some tickets for me to claim? Is adding to API docs more important than updating the examples and howtos? It looks like to add to the API docs you just update the doc strings for functions and let someone let pydoctor do its magic later. Is that true? Is there a special Twisted pydoctor incant to see how they'd look on the web before doing any hasty patch-submitting? I'd like to update http://twistedmatrix.com/trac/wiki/ContributingToTwistedLabs to include some of this information as well as help I got earlier for updating xhtml documentation. Thanks, -Jessica
![](https://secure.gravatar.com/avatar/7ed9784cbb1ba1ef75454034b3a8e6a1.jpg?s=120&d=mm&r=g)
On Thu, 30 Jul 2009 20:49:54 -0400, Jessica McKellar <jessica.mckellar@gmail.com> wrote:
Hi Jessica, First let me say hooray. :) Any attention you can direct at any of Twisted's documentation will be much appreciated. :) Don't worry about trying to find an unassigned ticket. We try to keep all tickets assigned to someone. That doesn't mean the person is actively working on resolving the ticket, just that at some point in time, someone felt like that assignee should have responsibility for it. You may want to comment on a ticket before starting to work on it, asking if anyone is really working on it, or if there is any work which has been started which you can pick up. However, for most tickets, you'll very likely find answers in the negative for both these questions.
Either API doc work or howto work will be appreciated. I find that the API docs are in better shape overall (except for a few dark corners), so I would probably recommend focusing on howtos (and perhaps examples). Here are a few tickets in particular which I think it would be worthwhile to resolve: http://twistedmatrix.com/trac/ticket/1136 http://twistedmatrix.com/trac/ticket/1627 http://twistedmatrix.com/trac/ticket/3784 http://twistedmatrix.com/trac/ticket/3835
Yep, that's right. There's just a little specialness involved in the pydoctor invocation. Here's a decent command line to use: pydoctor --quiet --make-html --system-class \ pydoctor.twistedmodel.TwistedSystem --add-package twisted where the final "twisted" is the path to the "twisted" directory beneath a Twisted checkout (ie, the path which has as a sibling the "doc" directory, the "README" file, etc). Make sure you have epydoc installed or pydoctor will give you unpleasant looking results with unparsed epytext strings in it. There are two other areas in which I think documentation work would yield a great benefit. First, in general editing for readability, coherency, and general ease of understandability of any of the existing documents. These are often rough first drafts which no one was interested in polishing, or rough first drafts which have had dozens of minor edits by multiple authors with little concern for the overall document flow. Second, in a comprehensive high-level overview of what documentation is missing, what documentation exists, and how it all fits together. The output of this would be recommendations about what new documentation to write, what old documentation to scrape, what documents could be merged, more closely linked, or otherwise made to cooperate to provide easier access to the information they present. This is probably a big task, and one which requires familiarity with a lot of Twisted. I'm not sure how much Twisted experience you already have, but I wanted to mention it in case it sounds like something you'd be interested in. Finally, a couple years ago an attempt was made to seriously do something about the state of the documentation. It didn't get far enough to produce actual changes to the documentation, but it did produce some output. This may be useful. You can find it on the wiki: http://twistedmatrix.com/trac/wiki/DocumentationAnalysis
That sounds like a great idea. Wiki pages require a special permission to be set on your trac user. If you let me know what your username is, I can take care of adding that for you. Thanks! Jean-Paul
![](https://secure.gravatar.com/avatar/46299857e4644e581677e2b80811b0c4.jpg?s=120&d=mm&r=g)
On Jul 30, 2009, at 11:35 PM, Jean-Paul Calderone wrote:
Traceback (most recent call last): File "/srv/trac/.local/lib/python2.5/site-packages/trac/web/ api.py", line 382, in send_error 'text/html') File "/srv/trac/.local/lib/python2.5/site-packages/trac/web/ chrome.py", line 475, in render_template return stream.render(method, doctype=doctype) File "/srv/trac/.local/lib/python2.5/site-packages/genshi/core.py", line 146, in render output = u''.join(list(generator)) File "/srv/trac/.local/lib/python2.5/site-packages/genshi/ output.py", line 200, in __call__ for kind, data, pos in stream: File "/srv/trac/.local/lib/python2.5/site-packages/genshi/ output.py", line 486, in __call__ for kind, data, pos in chain(stream, [(None, None, None)]): File "/srv/trac/.local/lib/python2.5/site-packages/genshi/ output.py", line 436, in __call__ for kind, data, pos in stream: File "/srv/trac/.local/lib/python2.5/site-packages/genshi/core.py", line 207, in _ensure for event in stream: File "/srv/trac/.local/lib/python2.5/site-packages/genshi/core.py", line 207, in _ensure for event in stream: File "/srv/trac/.local/lib/python2.5/site-packages/trac/web/ chrome.py", line 478, in _strip_accesskeys for kind, data, pos in stream: File "/srv/trac/.local/lib/python2.5/site-packages/genshi/ filters.py", line 313, in __call__ for kind, data, pos in stream: File "/srv/trac/.local/lib/python2.5/site-packages/genshi/ template.py", line 1145, in _match content = list(content) File "/srv/trac/.local/lib/python2.5/site-packages/genshi/ filters.py", line 313, in __call__ for kind, data, pos in stream: File "/srv/trac/.local/lib/python2.5/site-packages/genshi/ template.py", line 1120, in _match for event in stream: File "/srv/trac/.local/lib/python2.5/site-packages/genshi/ template.py", line 1109, in _strip event = stream.next() File "/srv/trac/.local/lib/python2.5/site-packages/genshi/ template.py", line 941, in _eval for event in substream: File "/srv/trac/.local/lib/python2.5/site-packages/genshi/ template.py", line 920, in _eval ctxt): File "/srv/trac/.local/lib/python2.5/site-packages/genshi/ template.py", line 930, in _eval result = data.evaluate(ctxt) File "/srv/trac/.local/lib/python2.5/site-packages/genshi/eval.py", line 101, in evaluate {'data': data}) File "/home/trac/.local/share/trac/templates/error.html", line 36, in <Expression u"get_reporter_id(req)"> <input type="hidden" name="reporter" value="${get_reporter_id (req)}" /> File "/srv/trac/.local/lib/python2.5/site-packages/trac/util/ __init__.py", line 50, in get_reporter_id name = req.session.get('name', None) AttributeError: 'NoneType' object has no attribute 'get' Yikes... S
![](https://secure.gravatar.com/avatar/fcdfff68a2c9b2d1d199e4626998c791.jpg?s=120&d=mm&r=g)
On Thu, Jul 30, 2009 at 10:35 PM, Jean-Paul Calderone <exarkun@divmod.com>wrote:
Let me second that Hooray! I thought I would jump in here with sort of an outsiders perspective on what I think needs to happen with the Twisted docs. Even though I've been using Twisted for a couple of years, I'm still new enough at it that I remember what my frustrations were (and sometimes still are) with the docs. This is all just my opinion, though...I don't mean to ruffle anyone's feathers.
I agree. The API docs need some work (there are a LOT of undocumented methods), but overall I think they are in much better shape than the rest. Howtos and examples would be fantastic, as well as clean up of some of the existing stuff (as Jean-Paul notes below). Browsing through the source code for long enough should give most coders the idea that all the pieces are there to build, e.g. an IMAP server, and those pieces ARE there, but it takes a lots of reading of lots of stuff in lots of different places (text docs, API docs, cred docs, application docs, etc., etc.) to figure out which pieces you need and how to put them together. I picked an IMAP server as an example, because of the AHA! moment I had when I read the following in the in the Abe Fettig O'Reilly book" "To make an IMAP server, write classes to implement the IAccount, IMailbox, IMessage, and IMessagePart interfaces defined in twisted.mail.imap4" The next two sentences complete the description of the process, but the key point is that for most parts of Twisted, there isn't much that tells you how all of the various parts fit together. And Twisted is BIG. It's hard to find things sometimes... Some of this could be improved by expounding on the API docs for these interfaces, but I think a broad overview for many of these areas would help immensely.
More true in some places than others, but a definite problem in some places. A personal pet peeve of mine is the "What Deferreds don't do: make your code asynchronous" section of http://twistedmatrix.com/projects/core/documentation/howto/gendefer.html There's a nice big example of what Deferred's don't do. OK, great, but if I'm new to Twisted and the async style of coding in general, then what *DO* I do to make my code asynchronous? (NOTE: some of this I think will be helped in the upcoming Deferred documentation rewrite...see the "Deferred documentation rewrite" thread on this list for the last day or two)
Emphasis on "how it fits together", IMHO. Navigating Twisted's' documentation (outside the API docs) is a nightmare. I often find myself just having to use Google searches.
I looked at this a while back, but I found it pretty much incomprehensible.
Thanks!
Jean-Paul
A couple of other things: There are several barriers to improving Twisted's docs - you have to know a fair amount about Twisted to write about some parts of it - you have to know a lot about Twisted to write about Twisted from a high level, big-picture sort of perspective - you have to know some seriously deep Twisted magic to even speak intelligibly about certain things - once you know enough about Twisted to really explain it...it seems you are no longer able to explain it :) Not trying to offend...everyone in the Twisted community I've talked to has been very helpful, but sometimes when you know a complex topic really well, it becomes difficult to explain things in such a way that someone new to that topic will understand. Does this make sense? Sometimes I think what is needed is someone who knows basically nothing about Twisted to go about learning it...and then writing down how they did it. Also: Some (more) documentation about how to write docs for Twisted would really help people who are willing (and in some cases eager!) to contribute, but feel the barrier to entry is too high (they don't want to figure out pydoctor + epydoc + xml processing + ...). Anyway, I hope my rambling is helpful... Kevin Horn
![](https://secure.gravatar.com/avatar/53cf3ca941974336efe1921cdef8e83b.jpg?s=120&d=mm&r=g)
----- Original Message ----- From: Kevin Horn "Sometimes I think what is needed is someone who knows basically nothing about Twisted to go about learning it...and then writing down how they did it." I'm trying to do this, by keeping notes as I learn twisted to build a custom email system. It's hard to admit how dumb I feel sometimes though. Frequently what happens is I get stuck when something doesn't work, and while searching through the docs I stumble upon something I already read earlier and thought I understood, and in re-reading it I get an insight or better understanding of what the doc means now that I can connect it to a specific problem and I have more experience. That's not usually how I use or experience documentation. Generally I use documentation to first find out what the tool is able to do, what it's supposed to be good for and good at, and then when I want to exploit one of those things I drill down to the specific instructions for it. That's hard to do as the docs are now, proably because the specific instructions depend so much on other specific instructions you may not know about yet. -Dave
![](https://secure.gravatar.com/avatar/ca0bbb8ddab6982c5536b7a07b85c1e6.jpg?s=120&d=mm&r=g)
Hi Jessica! I'm quite a newbie at twisted yet, and a newcomer to the project, so I do think that twisted documentation is one of it's more problematic areas!. My suggestion would be that more than *adding* documentation, I would suggest to first search-and-destroy deprecated documentation. For me, one of the most frustrating parts of using twisted was reading through the twisted core doc, trying to do things and failing, and then accessing #twisted and get people tell me thinks like 'oh, tap/tac/tml/mktap are no longer used', or things like that (I don't yet know which ones are supposed to be used and which don't). I understand that *rewriting* outdated doc is a major task, that's why I just suggest deleting it ;) (maybe in a new Revised Twiste Doc section of the web site or the like), then it probably would be clearer where to start and what needs most work. It could also be a good idea to try to make it more wiki-like at the beginning; I think there's quite a lot of people that would like to contribute with this, but having different documents/styles for different areas of the doc is not good, a wiki like doc might allow some people to work on giving a more cohesive look to the whole doc, and others to add the meat and bones, provide examples, etc, while allowing everyone to review it easily. Thinking about it, of course a doc on SVN would help in the same way ;). Truth is that I really don't know which are twisted best practices at the time (what I do works, but I don't know if it's the best way to do it), and checking outdated doc just confuses things even more... better no doc at all. I stop my suggestions here before someone yells me 'then do it!' :). -- Santiago
![](https://secure.gravatar.com/avatar/dd1243740a09f0676ef225404105cfc0.jpg?s=120&d=mm&r=g)
On Jul 31, 2009, at 10:19 AM, Santiago Aguiar wrote:
I'd like to second this, for at least the third or fourth time over the last several years ;-) To respond in advance to previous criticisms of the idea: * wiki syntax sucks * it doesn't have an acceptable form of version control * it's frustrating that it can't be easily bundled with a release tarball * the same info is on the mailing list, just use google to find it All 100% true. Nevertheless, I say wiki. It would provide the least complicated route for new users to document things they discover during the learning process, which is *exactly* the phase that is so hard for the more experienced users to properly explain. Hopefully over time the various technical writers who volunteer often could rewrite this content into the formal collection that is distributed with Twisted proper, but in the meantime, a community- edited wiki would also provide another window into the community for new developers. I also would recommend MediaWiki. Yes, less than perfect, not as good as this, that, or the other wiki. In the interest of keeping the bar for entry low, however, we would be harnessing a familiarity with MediaWiki provided by exposure to Wikipedia. Pragmatism, not idealism, and all that... -phil
![](https://secure.gravatar.com/avatar/fcdfff68a2c9b2d1d199e4626998c791.jpg?s=120&d=mm&r=g)
On Fri, Jul 31, 2009 at 9:34 AM, Phil Christensen <phil@bubblehouse.org>wrote:
Well, if there *were* to be a change in the documentation system used, I would push for moving to Sphinx, rather than a wiki. * it's rapidly becoming a "standard" for docs in the Python world * it has lots of neat features * it can be version controlled * multiple output formats (html, chm, Latex(and therefore PDF), etc.) * I just like it :) Just sayin' Kevin Horn
![](https://secure.gravatar.com/avatar/f4b759bb1fd0b5d37695b7de428ae9c7.jpg?s=120&d=mm&r=g)
Hi All,
Rather than change the documentation system entirely, who don't we just create a new resource for Twisted documentation - a Twisted Documentation Wiki. At the very least it can be a repository of various tips and tricks we happen to find, and maybe it can even be a staging ground for documentation that makes it into Twisted proper. Reza -- Reza Lotun +44 (0)7521 310 763 rlotun@gmail.com
![](https://secure.gravatar.com/avatar/ca0bbb8ddab6982c5536b7a07b85c1e6.jpg?s=120&d=mm&r=g)
Reza Lotun wrote:
Even if I re-suggested the wiki based documentation, I think it's important to be extra careful on how it's used. One thing I personally hate is projects whose documentation is basically wiki-based, and what you end having is a disconnected set of tips, many out of date, of how to do this and that. It could be OK it it's labeled 'Tipi-wiki' but not 'Documentation' :). As you say, I think that the wiki could be useful as a staging ground to build a formal documentation, but it shouldn't be even suggested for new users, since what they will probably find are many unfinished ideas. Alternatively, a separate doc repo with sphinx based doc could be built so that it will allow for collaborative development making clear that is a work in progress and a product with 'releases'. I say a different repo to avoid having to give commit access to code for people who are working on doc, maybe same repo with different permissions would be better. And I suggest sphinx to a) start from scratch and add existing doc as we see it's relevant, b) have a more flexible base than HTML docs, and for the reasons mentioned by Kevin (plus I want to learn it :P). -- Santiago.
![](https://secure.gravatar.com/avatar/f4b759bb1fd0b5d37695b7de428ae9c7.jpg?s=120&d=mm&r=g)
I agree - the wiki shouldn't *replace* the documentation, but the reality is I have loads of bookmarks of blog posts and discussions on the mailing list, and it'd be nice if I could to go one place to find all that type of info. A "recipe" or "cookbook" wiki might be all we need, with the ability to comment on each. The Activestate Python Cookbook is kinda what I'm thinking about: http://code.activestate.com/recipes/langs/python/
I haven't yet looked at lore or sphinx, but is there a way to support sphinx via lore? Reza -- Reza Lotun +44 (0)7521 310 763 rlotun@gmail.com
![](https://secure.gravatar.com/avatar/dd1243740a09f0676ef225404105cfc0.jpg?s=120&d=mm&r=g)
On Jul 31, 2009, at 12:32 PM, Reza Lotun wrote:
I agree with all of the above. A wiki is nice when there is no suitable formal documentation available for a topic. I think I may have used the aphorism once before, "documentation is like sex, when it's good, it's great, when it's bad it's still better than nothing." (no offense intended to any with delicate sensibilities, btw ;-) My only question about Sphinx, isn't it just for API docs? Also, can it interpret Zope interfaces like pydoctor can? Personally I'm pretty happy with the API docs (although there's always room for improvement in the actual docstrings), I think if there's a documentation need that's more dire, it's the long-form instructional kind. I just don't want to sidetrack *that* discussion by getting into API documentation concerns. -phil
![](https://secure.gravatar.com/avatar/fcdfff68a2c9b2d1d199e4626998c791.jpg?s=120&d=mm&r=g)
On Fri, Jul 31, 2009 at 11:52 AM, Phil Christensen <phil@bubblehouse.org>wrote:
My only question about Sphinx, isn't it just for API docs? Also, can it interpret Zope interfaces like pydoctor can?
Sphinx is great for writing all sorts of docs. At its core, it's basically a system for gluing together a bunch of RestructiredText documents and automatically cross-linking them, building an index, a javascript-based search capability, handling tables of contents, and outputing the results in various formats. But then it's extensible...so you can write plugins for it. One of the earliest (and included with the base package) is the "autodoc" extension which makes generating nice clean API docs from your docstrings. What it does _not_ do is process all those epydoc bits that are in Twisted's docstrings, though you might be able to write an extension for that. It expects docstrings to be in RestructuredText. AFAIK Sphinx does not have support for Zope interfaces, but I haven't looked in a while, and I seem to remember that someone was working on adding it...though I may be making that up. At any rate, I think using Sphinx for Twisted's API docs (at this juncture, anyway) would be a bad idea. But Sphinx excels at the kind of "long-form, instructional" docs you mention below. I use it for all my PHP projects (which of course, the autodoc extension doesn't work on), and I'm really happy with it.
Agreed.
I just don't want to sidetrack *that* discussion by getting into API documentation concerns.
Double Agreed.
-phil
Kevin Horn
![](https://secure.gravatar.com/avatar/fcdfff68a2c9b2d1d199e4626998c791.jpg?s=120&d=mm&r=g)
On Fri, Jul 31, 2009 at 12:11 PM, Kevin Horn <kevin.horn@gmail.com> wrote:
OK, having Googled around, it seems I was *not* making this up. There is some support for Zope interfaces in Sphinx, though it looks pretty minimal at the moment. Just, you know, in case anyone was interested. :) Kevin Horn
![](https://secure.gravatar.com/avatar/2326f5e54bb073d5252a23b39708b839.jpg?s=120&d=mm&r=g)
A lot has been said in this discussion but I'll add one thing. On 07/31/2009 06:32 PM, Reza Lotun wrote:
What I miss is a comment system related to the API docs so that it is possible for beginners and others to add small notes there without having to make a ticket, wait for it to be assigned etc. Apart from that I will just welcome anyone who writes documentation. Kind regards, Tarjei
-- Tarjei Huse Mobil: 920 63 413
![](https://secure.gravatar.com/avatar/95658668bebdc20db10bb5ccc1603017.jpg?s=120&d=mm&r=g)
2009/8/3 tarjei <tarjei@nu.no>:
I have an old, half finished idea in this area, which is that pydoctor has a mode which runs a web server and allows you to edit the docstrings online. It needs a bunch of work to get to useful -- probably user accounts, a way of not losing all changes when you restart the server, a lot of polish and, indeed, maybe a comment feature or discussion view -- but I think the idea is sound. If you have pydoctor installed, you can try pydoctor --auto in a directory containing some python modules. Cheers, mwh
![](https://secure.gravatar.com/avatar/fcdfff68a2c9b2d1d199e4626998c791.jpg?s=120&d=mm&r=g)
On Fri, Jul 31, 2009 at 11:14 AM, Santiago Aguiar <santiago.aguiar@gmail.com
wrote:
In my experience, this what happens to every project that a) uses a wiki for core documentation and b) has more than one person editing the wiki.
I'd love to see a documentation "reboot" using Sphinx, but not if it's going to be a half-baked, never-finished project. Since Jessica started this thread, and is the only person (so far) who has given any sort of commitment to actually working on new/revised docs, I'd really like to hear her opinion. Jessica? You still out there? Hope we haven't scared you off... I'd also be interested in hearing the opinions of some of the core Twisted guys on the various things we've been talking about here. What do you guys think about using a different docs system than what is being used now? If you guys are all dead set against it, there's not much point hashing out the details... Also, what do the Twisted core devs think about having a secondary wiki/cookbook thingy outside of the core docs? Kevin Horn
![](https://secure.gravatar.com/avatar/7ed9784cbb1ba1ef75454034b3a8e6a1.jpg?s=120&d=mm&r=g)
On Fri, 31 Jul 2009 11:55:08 -0500, Kevin Horn <kevin.horn@gmail.com> wrote:
I don't find that people trying to use Twisted complain about the presentation of the documentation. I find that they complain about its content. So, I think that it's the content that needs to be addressed. I don't *think* that switching to Sphinx (or anything else) is going to make any different to the content of the current documentation. One might argue that Lore is a significant barrier to entry for new contributions to the documentation, but you'd have to try pretty hard to convince me. Pretty much anyone can write some simple (Lore x)html. And if they can't, then there are other people willing to translate plain text into Lore input documents.
Also, what do the Twisted core devs think about having a secondary wiki/cookbook thingy outside of the core docs?
As a staging area for development of future core docs, I think I would recommend using a version control system (perhaps a distributed one), not a wiki. As something intended to be user-facing, I don't think it's a great idea. Of course, there *is* a wiki hosted on the website already... And it has some documentation on it... So what's being proposed, exactly? :) Obviously I'm speaking for myself, not for all the other people who have contributed to Twisted. Jean-Paul
![](https://secure.gravatar.com/avatar/dd1243740a09f0676ef225404105cfc0.jpg?s=120&d=mm&r=g)
On Jul 31, 2009, at 1:11 PM, Jean-Paul Calderone wrote:
What I was suggesting originally was that a wiki would be a staging ground for those core docs. IMO, traditional version control is not generally well-suited for collaborative document development. I guess what I'm thinking is, no one likes writing docs, so when we have to good fortune to encounter someone who does, let's make it as easy as possible.
True. I guess there's only two considerations. 1. Using the current Trac installation for a user-edited wiki might convey that the contents are endorsed by Twisted proper. 2. Although I've become fairly used to it, Trac's wiki isn't the greatest, and more importantly, Trac itself sometimes seems to be held together with bubble gum and scotch tape. -phil
![](https://secure.gravatar.com/avatar/fcdfff68a2c9b2d1d199e4626998c791.jpg?s=120&d=mm&r=g)
On Fri, Jul 31, 2009 at 12:11 PM, Jean-Paul Calderone <exarkun@divmod.com>wrote:
True enough. And in my case, at least, navigation.
I don't think using Lore is a problem particularly...but it need to be documented more clearly, and all in one place. Right now, there's stuff on the tracwiki, in the main docs, on the mailing list, etc. Particularly, the answers to the questions: - How do I build the documentation using Lore? - What is the Lore xhtml syntax, and how should I use it? - What is the process for making a documentation contribution? Please note, that I was originally suggesting Sphinx primarily as an alternative to a wiki-based system, rather than as a replacement for Lore...but the conversation kind of mutated. :)
Agreed, wiki = yuck (for this). Even as a "staging ground". IMO trac-wiki is really only suited for "marketing" type content.
![](https://secure.gravatar.com/avatar/dd1243740a09f0676ef225404105cfc0.jpg?s=120&d=mm&r=g)
On Jul 31, 2009, at 1:34 PM, Kevin Horn wrote:
Okay, tell me again what exactly the problem is with a wiki? I feel like the same thing happens every time we discuss documentation. Someone makes a recommendation to do it the easy way, and someone else dismisses any solution that doesn't satisfy their programmer's OCD. Yes, I'd love to see version controlled XML documentation that adheres to a common writing style that is enforced across the board, but this talk has been happening for *YEARS* and there has been little improvement of significance (I have to emphasize that I understand a number of people have worked very hard on this, and I don't mean to denigrate their contributions). People keep telling me wikis are bad, but I'm still not getting the 'why' -- I just hear "wikis are bad for documentation" repeatedly presented as a fact. They do seem to work reasonably well for scores of other projects. PRAGMATISM!!! ;-) -phil
![](https://secure.gravatar.com/avatar/fcdfff68a2c9b2d1d199e4626998c791.jpg?s=120&d=mm&r=g)
On Fri, Jul 31, 2009 at 12:57 PM, Phil Christensen <phil@bubblehouse.org>wrote:
Well, pragmatism is my basic reason for not liking wikis for docs. :) For me, it boils down to every time I've worked on or with a project that used wikis for docs (assuming that the project is of at least moderate size and has more than a couple of editors) that documentation has been terrible. It's ended up as a disjointed mess, that's hard to navigate, and has a bajillion half-completed thoughts, and lots of outdated cruft. Of course, some could say the same of (at least parts of) the current Twisted documentation. ;) It's not that there's anything wrong with wiki's per se, it's just that they encourage "bad habits". If there were a solid editorial process in place, where someone was specifically responsible for reviewing, editing, splitting, merging, and correcting documents, then a wiki could probably work. But I think it's easier to get coherent docs using tools that encourage "good habits". Obviously my definitions of "good habits" and "bad habits" are pretty vague...and not entirely spelled out even in my own mind. Er, sorry about that... Kevin Horn
![](https://secure.gravatar.com/avatar/dd1243740a09f0676ef225404105cfc0.jpg?s=120&d=mm&r=g)
On Jul 31, 2009, at 2:15 PM, Kevin Horn wrote:
Indeed; I wouldn't use those particular words because of their pejorative qualities, but I think the end result is that way because of the level of interest in maintaining documentation for a particular project. That's a different issue, but it is the reason why I'd like the documentation process to be easy for 'regular developers' (e.g., not the proto-deities that make up the core devs ;-) to contribute to.
Well, honestly, I don't disagree with you at all. A good set of documentation requires a great deal of work, and the amount of rewriting and fact-checking required is proportional to the amount of development going on. Twisted development has slowed somewhat over the years in a few areas, but it still moves quite quickly in others. I don't see any documentation system making this go away. It will always have to be edited and revised, and it will often be out of date. I think the issue is one of not 'scaring away' potential writers with a cumbersome process. Correct me if I'm wrong, but the current process for writing Twisted documentation is as follows (at least for people without commit access): 1. check out a directory of Lore XHTML files 2. make my changes or create a new file, making sure not to work against the sparsely documented schema ;-) 3. generate a patch 4. create a ticket, attach the patch 5. bug someone on #twisted or the mailing list to do a review 6. wait some unspecified amount of time 7. code is merged into trunk 8. wait for the live docs to be regenerated Compared to a wiki: 1. create an account (let's assume it's email-verified) 2. write or edit some docs 3. see your docs live immediately Which do you think is going to be a more compelling process for people to get involved with? The first process is straight from the software development playbook, so it seems like a good idea, but I don't agree with the need for the same regimented process to exist w.r.t. documentation, especially when we are in such dire straights in that regard. I'm not really married to the idea of a wiki per se, it's just that it seems to be the closest thing that provides contributors a low bar for entry and instant gratification by seeing their work online (and on a list of recent changes). Plus, it addresses a concern Ying Li just posted as I was writing this: you automatically get revisions, last- modified dates, and diffs of those revisions. Yes, some of this documentation will not be of the quality we would like it to be. But it will still exist, which is a hell of a lot better than nothing. Plus, it gives the technical writers we occasionally see interest from some source material to work with in the first place, which has got to be helpful towards the ultimate goal of formally written core documentation. -phil
![](https://secure.gravatar.com/avatar/a7267ede9d6ea0117c5438c398a1a212.jpg?s=120&d=mm&r=g)
I am not scared off. In fact, I'm updating the wiki/ContributingToTwistedLabs and related how-to-edit-documentation pages right now. I think adding content to and restructuring the howtos gives me a full plate already, and to be honest I'm more interested in the content than the presentation. People who have more experience with the successes and failures of the various ways to organize project content online can make that decision and I'd happily help move content over to another format should that be necessary. Thanks for all the help and feedback, everyone. -Jessica On Fri, Jul 31, 2009 at 12:55 PM, Kevin Horn <kevin.horn@gmail.com> wrote:
![](https://secure.gravatar.com/avatar/fcdfff68a2c9b2d1d199e4626998c791.jpg?s=120&d=mm&r=g)
On Fri, Jul 31, 2009 at 12:16 PM, Jessica McKellar < jessica.mckellar@gmail.com> wrote:
Great! I think that this is a critical first step.
Fantastic. Content is definitely more important, and as this thread proves, its easy to get bogged down in the "how" rather than the "what".
Kevin Horn
![](https://secure.gravatar.com/avatar/e1554622707bedd9202884900430b838.jpg?s=120&d=mm&r=g)
On Fri, Jul 31, 2009 at 1:16 PM, Jessica McKellar < jessica.mckellar@gmail.com> wrote:
I see the edits in the timeline. They look great! Thanks! :-D Do you have any ideas for improving the front page as well? I think the random-list-of-links approach to its information architecture hasn't been terribly effective. I think adding content to and restructuring the howtos gives me a full plate
already, and to be honest I'm more interested in the content than the presentation.
This is good to hear. I think it should definitely be our first priority. Thanks for all the help and feedback, everyone.
Please don't hesitate to ask for any other help or feedback you might need. People who have more experience with the successes and failures of the
various ways to organize project content online can make that decision (...)
I don't want to stifle the possible discussion of using a different system, but for those of you interested in switching to Sphinx or something else, there are several things you need to address: First and foremost, you need to make a commitment to *maintaining* the documentation infrastructure. This includes teaching other folks how to use it. Second, you need to submit a complete proposal. This means we need a clear enumeration of what benefits the different documentation technology will have; this needs to be written up in one place so that we can evaluate it and discuss it without chasing down a million little email threads or ticket comments. Such a proposal also needs to address integration of automated testing and automated doc-building with the examples. You'll need to set up a builder that will replace our current documentation builder. You'll need to update all the relevant wiki pages. Third, you need to address the transition plan. Just translating all the documentation by hand is a possibility, but it has potential problems. What happens if we discover a problem with the new tool? How do we roll back? and I'd happily help move content over to another format should that be
necessary.
Lucky for you, Jessica appears to have volunteered to do this for you :-). (You might want to confirm that first, though.) I certainly wouldn't mind switching to a tool that has lots of fancy features that lore lacks, but a hit-and-run approach where we just switch tools in the hopes that it will make something better may leave us in a worse situation than we already are.
![](https://secure.gravatar.com/avatar/46299857e4644e581677e2b80811b0c4.jpg?s=120&d=mm&r=g)
On Jul 31, 2009, at 4:23 PM, Glyph Lefkowitz wrote: ...
I'm wondering if there's an inventory of the various types of documentation systems currently in use. Not just "systems" but mostly "stray documentation." From this discussion, it doesn't seem that the Lore formatted and built docs are as much of a problem as the fragmentation into tracwiki, main docs, mailing list, etc. that we've discovered as we've been discussing it. Since "everything" is not in the build system anyway, perhaps starting a branch, in a new build system (Sphinx), where we pull things in, one chunk at a time, will not be a hit and run approach, but will force the reorganization of the docs into one, actually unified format. Presentation is another issue, for another day. Then we can all see what's documented, what's not, what's old and out of date, etc. With a dating system as discussed elsewhere, the docs will actually improve over time instead of becoming more fractured and out of date with no traceability. Just my 13.4 cents. S
![](https://secure.gravatar.com/avatar/e1554622707bedd9202884900430b838.jpg?s=120&d=mm&r=g)
On Fri, Jul 31, 2009 at 11:10 PM, Steve Steiner (listsin) < listsin@integrateddevcorp.com> wrote:
"This time for sure, Rocky!" If you actually want to volunteer to do all of the work for this (which I outlined in my previous message) then feel free. But attempts to fix the world by blowing it up are rarely successful. A better approach would be to incrementally enumerate the things which have been covered on the mailing list and wiki, then pull them into the lore docs one at a time. Of course, the people who have actually volunteered their time, rather than their suggestions, seem to agree with this general outline :). Something you have to keep in mind with grand efforts like "let's rewrite all the docs in the hottest new ReFrumpledText format" is that Twisted is a product mostly of people's spare time, and therefore the person working on it may suddenly become busy and lose interest. When they *do* lose interest — possibly for a year or longer — we need to make sure that things are in a good state in the meanwhile, and their efforts have improved things. Each small improvement to the lore documentation will improve the overall documentation situation. As you put it, a big problem here is: the fragmentation into tracwiki, main docs, mailing list, etc
if the fragmentation is instead into tracwiki, main docs, sphinx docs, mailing list, etc, the problem has actually gotten *worse*, not better. Especially if the sphinx person walks away halfway through the effort, and then some other person comes along and says, "oh hey, what we should *really * do is rewrite all the docs in YAML", then rewrites another small subset of them and leaves. I hope you can see why I want to hold on to our current toolchain until we have someone around who has demonstrated a much deeper commitment to documentation than anyone yet has. For example, Steve, if you close 100 existing documentation tickets in the next week, then make the exact same suggestion again, I'll be a lot less resistant ;-).
![](https://secure.gravatar.com/avatar/e1554622707bedd9202884900430b838.jpg?s=120&d=mm&r=g)
On Sat, Aug 1, 2009 at 12:11 AM, Steve Steiner < listsin@integrateddevcorp.com> wrote:
I do tend to use a lot of words to get things across. Sorry ;-).
Where would we put an "inventory" project for it to be most useful?
As JP mentioned earlier in this thread, some work has already taken place here: <http://twistedmatrix.com/trac/wiki/DocumentationAnalysis>. If you'd like to contribute to that, please let somebody on the #twisted IRC channel know your trac username so we can set you up with wiki-edit permission.
![](https://secure.gravatar.com/avatar/46299857e4644e581677e2b80811b0c4.jpg?s=120&d=mm&r=g)
I get it, really.
I do tend to use a lot of words to get things across. Sorry ;-).
Duly noted ;-).
It doesn't really look like that effort went anywhere. On: http://twistedmatrix.com/trac/wiki/DocumentationAnalysis/DocumentList The first document is User reviewed by JoeBlow with Expert Reviews by JaneBlogs. There are only five more entries and only one is "completed" with a User and Expert review. What the heck that properly vetted doc has to do with anything is not particularly clear. Maybe we should have an auction? Auction: Document blah-de-blah Supplier will document blah-de-blah, work with Sponsors to incorporate blah-de-blah into their apps, and release the copyright to the resultant documentation and copyright to the Twisted Software Foundation. Supplier will provide up to X hour of consulting to each of the first three Sponsors (Founding Sponsors) to help refine the documentation and make sure it can be used in real-world projects. Minimum Founder bid is $50. Minimum Contributor bid is $35. Sponsor1: I'll give $50 for someone to explain blah-de-blah and how to use it in my app Sponsor2: I'd give another $50 if I could get this working in my app Supplier1: I'd have to have $200 to take this on. I'll also offer discounted consulting on the same topic at $??/hr. to anyone who contributes $35 or more to this documentation project after Sponsor3: I'll chip in the extra $100 if I can be one of the first three. I'm really stuck on blah-de-blah and could use some help. Moderator: Ok, we seem to have a deal. Sponsor 1 is in for $50, Sponsor 2 is in for $50, Sponsor 3 is in for $100. Each of them receives X hours of help from Supplier 1 which will be used to advance their projects and improve the documentation. Any future Contributors, over $35, receive discounted consulting from Supplier1 at $??/hr. for any work related to this issue as well as access to beta versions of the documentation and early-release versions of the sample code. Please sign off on this agreement and submit your payment. Supplier1 agrees to provide a first draft within X-period-of-time. Consulting arrangements begin then. The Twisted Software Foundation accepts a donation of 10% of all transactions for facilitating these transactions and to generally advance Twisted. (http://twistedmatrix.com/trac/wiki/TwistedSoftwareFoundation ) Just a thought... S P.S. I apologize in advance for any formatting issues with the above. Obviously, I'm not following the coding spec and pylint barfed on the whole thing... ;-). s
![](https://secure.gravatar.com/avatar/e1554622707bedd9202884900430b838.jpg?s=120&d=mm&r=g)
On Sat, Aug 1, 2009 at 11:30 PM, Steve Steiner (listsin) < listsin@integrateddevcorp.com> wrote:
I don't think it's quite fair to say it didn't go *anywhere*, it just didn't get anywhere particularly complete. I think the reviews that have been done thus far are still a valuable resource for someone interested in working on the documentation.
Maybe we should have an auction?
(snipped lengthy description of labor-intensive bounty resolution process) What we really need is for people in the community to knuckle down and do the work of actually reviewing the documents, rewriting things, soliciting feedback, copy-editing the documents, checking for accuracy, etc. As Jessica did when she started the thread :). Frankly, the work is boring: what the documentation needs is not a dynamic market-driven system for managing contributions and paying bounties, it's not a comprehensive overhaul of our entire documentation toolchain. What it needs is for people to sit down and read all the documentation, make a list of all of its problems, one at a time, and then write, delete, and edit as necessary, incorporating feedback from the community. Once we have people taking part in that effort and working with the docs on a daily or weekly basis, *then* we can listen to the reality of the problems and frustrations that they have working with the tools or determining priorities or whatever and solve those problems. Separately from that, if you're interested in contributing some time to manage a bounty program for the TSF, that might be a reasonable idea. Other projects have experimented with bounty-based systems for resolving bugs. Unfortunately there are a number of difficulties with those systems, which are really out of the scope of this discussion, but they're worth talking about if you have a real interest in doing it.
![](https://secure.gravatar.com/avatar/46299857e4644e581677e2b80811b0c4.jpg?s=120&d=mm&r=g)
On Aug 2, 2009, at 12:41 AM, Glyph Lefkowitz wrote:
Ok, fine, it went "over there." Now...if you're just starting out, how much does "over there" help and how much has "over there" ever been integrated into anything else? How would you even find "over there?" I guess I'm still frustrated with the lack of an overarching doc organization scheme. I've tried to use various parts of the Twisted eco-system, on a number of projects, and have repeatedly gotten stuck on outdated, disjointed, disorganized, and just plain wrong documentation. As I've said before, the complete lack of date/version stamping on example code has been the greatest source of frustration. I've, more than once, gotten half-way through a tutorial and realize that it just plain didn't work with the current version. End of exploration. I used the best result I found in my search and it just didn't work. Gotta move on; client's getting annoyed. Unbillable time; bad.
Hasn't happened, and isn't going to happen, in my opinion, until there's some actual incentive for "people in the community", which is a small subset of the "potential community", to actually do anything about it. Documentation is not needed by the "community" as much as it's needed by the "potential community."
Frankly, the work is boring and it ain't happening, never has, and never will unless something changes. Bounty system, torture, Target coupons, whatever...
See above. How many years does the same thing have to happen for there to be a realization that, without a fundamental change, nothing is going to happen? If you paid someone, 20 hours a week, to babysit the documentation effort, they would. Depending on their competence, there would be progress. If you wait, endlessly for "Once we have people taking part in that effort and working with the docs on a daily or weekly basis", you're going to get the same result.
I'm not sure what I'm interested in, at this point, but I certainly am not interested in doing it for free and neither is anyone else. In my opinion, and by existing evidence, coherent documentation ain't happening, and won't, unless something fundamental changes. It's not my project and it's not for me to decide but it seems to me that waiting for "people to take part in that effort" is doomed to waiting forever... S
![](https://secure.gravatar.com/avatar/307ebad5fc7824b9f223fbad5698d278.jpg?s=120&d=mm&r=g)
On Fri, Jul 31, 2009 at 10:19 AM, Santiago Aguiar<santiago.aguiar@gmail.com> wrote:
Something that I find irritating about a lot of documentation that I read is that it is not dated. All the twisted docs are labeled with a version (currently 8.2.0) but this is meaningless to me. I think a big improvement would be putting in a small bit of context around the documentation , such as: * when the howto/tutorial was last updated * what version (of deferreds/imap/whatever the howto/tutorial is about) the howto/tutorial was written for What would be even more useful, although it would also be a lot more work, is to perhaps link to a diff between the code as it is now and as it was when the tutorial was written. Or just a changelog. Also, I oppose deleting outdated documentation outright, as I think limited information is better than none. However, I think it should be clearly marked as such, so readers understand that the doc may not give them correct instructions. (This does not apply to the API docs as since the API docs are automatically generated, they should always current given the Twisted coding standards.)
![](https://secure.gravatar.com/avatar/46299857e4644e581677e2b80811b0c4.jpg?s=120&d=mm&r=g)
On Jul 31, 2009, at 3:37 PM, Ying Li wrote:
That is one of my pet peeves about project documents of any type. Tutorials, and examples especially. There's nothing more frustrating than getting 1/2 way through a tutorial, only to realize that the rest of it is just not going to work with the current version and that all that effort was wasted because that's not how it's done any more. If it's not dated, and perhaps tagged with the first version with which it was released, and the most recent version at which it was modified, it's impossible to figure out what's current, relevent, and likely to actually work with the current release. S
![](https://secure.gravatar.com/avatar/46299857e4644e581677e2b80811b0c4.jpg?s=120&d=mm&r=g)
On Jul 31, 2009, at 10:25 PM, Itamar Shtull-Trauring wrote:
Sorry for not being familiar with the overall system, but where would one put a suggestion regarding the overall date/time/version stamping of every document in the system? I'd be happy to propose a standard format but I'm not sure where to put it so it's universally applied (across all products). I know Jessica McKellar <jessica.mckellar@gmail.com> started working on the "ContributingToTwistedLabs" wiki how-to-edit-documentation docs. Can we get it in there, or where would be the best place to get this started? I've CC's Jessica to see how she thinks this might be integrated as well. Thanks, S AKA/ssteiner AKA/ssteinerX@gmail.com
![](https://secure.gravatar.com/avatar/307ebad5fc7824b9f223fbad5698d278.jpg?s=120&d=mm&r=g)
On Fri, Jul 31, 2009 at 10:25 PM, Itamar Shtull-Trauring<itamar@itamarst.org> wrote:
Could you file some bugs to that effect, so we don't forget these suggestions? Thanks!
Done: http://twistedmatrix.com/trac/ticket/3945 The formatting on the description is off though (I tried to make bullet points and failed), and I don't have permission to change it.
![](https://secure.gravatar.com/avatar/7ed9784cbb1ba1ef75454034b3a8e6a1.jpg?s=120&d=mm&r=g)
On Thu, 30 Jul 2009 20:49:54 -0400, Jessica McKellar <jessica.mckellar@gmail.com> wrote:
Hi Jessica, First let me say hooray. :) Any attention you can direct at any of Twisted's documentation will be much appreciated. :) Don't worry about trying to find an unassigned ticket. We try to keep all tickets assigned to someone. That doesn't mean the person is actively working on resolving the ticket, just that at some point in time, someone felt like that assignee should have responsibility for it. You may want to comment on a ticket before starting to work on it, asking if anyone is really working on it, or if there is any work which has been started which you can pick up. However, for most tickets, you'll very likely find answers in the negative for both these questions.
Either API doc work or howto work will be appreciated. I find that the API docs are in better shape overall (except for a few dark corners), so I would probably recommend focusing on howtos (and perhaps examples). Here are a few tickets in particular which I think it would be worthwhile to resolve: http://twistedmatrix.com/trac/ticket/1136 http://twistedmatrix.com/trac/ticket/1627 http://twistedmatrix.com/trac/ticket/3784 http://twistedmatrix.com/trac/ticket/3835
Yep, that's right. There's just a little specialness involved in the pydoctor invocation. Here's a decent command line to use: pydoctor --quiet --make-html --system-class \ pydoctor.twistedmodel.TwistedSystem --add-package twisted where the final "twisted" is the path to the "twisted" directory beneath a Twisted checkout (ie, the path which has as a sibling the "doc" directory, the "README" file, etc). Make sure you have epydoc installed or pydoctor will give you unpleasant looking results with unparsed epytext strings in it. There are two other areas in which I think documentation work would yield a great benefit. First, in general editing for readability, coherency, and general ease of understandability of any of the existing documents. These are often rough first drafts which no one was interested in polishing, or rough first drafts which have had dozens of minor edits by multiple authors with little concern for the overall document flow. Second, in a comprehensive high-level overview of what documentation is missing, what documentation exists, and how it all fits together. The output of this would be recommendations about what new documentation to write, what old documentation to scrape, what documents could be merged, more closely linked, or otherwise made to cooperate to provide easier access to the information they present. This is probably a big task, and one which requires familiarity with a lot of Twisted. I'm not sure how much Twisted experience you already have, but I wanted to mention it in case it sounds like something you'd be interested in. Finally, a couple years ago an attempt was made to seriously do something about the state of the documentation. It didn't get far enough to produce actual changes to the documentation, but it did produce some output. This may be useful. You can find it on the wiki: http://twistedmatrix.com/trac/wiki/DocumentationAnalysis
That sounds like a great idea. Wiki pages require a special permission to be set on your trac user. If you let me know what your username is, I can take care of adding that for you. Thanks! Jean-Paul
![](https://secure.gravatar.com/avatar/46299857e4644e581677e2b80811b0c4.jpg?s=120&d=mm&r=g)
On Jul 30, 2009, at 11:35 PM, Jean-Paul Calderone wrote:
Traceback (most recent call last): File "/srv/trac/.local/lib/python2.5/site-packages/trac/web/ api.py", line 382, in send_error 'text/html') File "/srv/trac/.local/lib/python2.5/site-packages/trac/web/ chrome.py", line 475, in render_template return stream.render(method, doctype=doctype) File "/srv/trac/.local/lib/python2.5/site-packages/genshi/core.py", line 146, in render output = u''.join(list(generator)) File "/srv/trac/.local/lib/python2.5/site-packages/genshi/ output.py", line 200, in __call__ for kind, data, pos in stream: File "/srv/trac/.local/lib/python2.5/site-packages/genshi/ output.py", line 486, in __call__ for kind, data, pos in chain(stream, [(None, None, None)]): File "/srv/trac/.local/lib/python2.5/site-packages/genshi/ output.py", line 436, in __call__ for kind, data, pos in stream: File "/srv/trac/.local/lib/python2.5/site-packages/genshi/core.py", line 207, in _ensure for event in stream: File "/srv/trac/.local/lib/python2.5/site-packages/genshi/core.py", line 207, in _ensure for event in stream: File "/srv/trac/.local/lib/python2.5/site-packages/trac/web/ chrome.py", line 478, in _strip_accesskeys for kind, data, pos in stream: File "/srv/trac/.local/lib/python2.5/site-packages/genshi/ filters.py", line 313, in __call__ for kind, data, pos in stream: File "/srv/trac/.local/lib/python2.5/site-packages/genshi/ template.py", line 1145, in _match content = list(content) File "/srv/trac/.local/lib/python2.5/site-packages/genshi/ filters.py", line 313, in __call__ for kind, data, pos in stream: File "/srv/trac/.local/lib/python2.5/site-packages/genshi/ template.py", line 1120, in _match for event in stream: File "/srv/trac/.local/lib/python2.5/site-packages/genshi/ template.py", line 1109, in _strip event = stream.next() File "/srv/trac/.local/lib/python2.5/site-packages/genshi/ template.py", line 941, in _eval for event in substream: File "/srv/trac/.local/lib/python2.5/site-packages/genshi/ template.py", line 920, in _eval ctxt): File "/srv/trac/.local/lib/python2.5/site-packages/genshi/ template.py", line 930, in _eval result = data.evaluate(ctxt) File "/srv/trac/.local/lib/python2.5/site-packages/genshi/eval.py", line 101, in evaluate {'data': data}) File "/home/trac/.local/share/trac/templates/error.html", line 36, in <Expression u"get_reporter_id(req)"> <input type="hidden" name="reporter" value="${get_reporter_id (req)}" /> File "/srv/trac/.local/lib/python2.5/site-packages/trac/util/ __init__.py", line 50, in get_reporter_id name = req.session.get('name', None) AttributeError: 'NoneType' object has no attribute 'get' Yikes... S
![](https://secure.gravatar.com/avatar/fcdfff68a2c9b2d1d199e4626998c791.jpg?s=120&d=mm&r=g)
On Thu, Jul 30, 2009 at 10:35 PM, Jean-Paul Calderone <exarkun@divmod.com>wrote:
Let me second that Hooray! I thought I would jump in here with sort of an outsiders perspective on what I think needs to happen with the Twisted docs. Even though I've been using Twisted for a couple of years, I'm still new enough at it that I remember what my frustrations were (and sometimes still are) with the docs. This is all just my opinion, though...I don't mean to ruffle anyone's feathers.
I agree. The API docs need some work (there are a LOT of undocumented methods), but overall I think they are in much better shape than the rest. Howtos and examples would be fantastic, as well as clean up of some of the existing stuff (as Jean-Paul notes below). Browsing through the source code for long enough should give most coders the idea that all the pieces are there to build, e.g. an IMAP server, and those pieces ARE there, but it takes a lots of reading of lots of stuff in lots of different places (text docs, API docs, cred docs, application docs, etc., etc.) to figure out which pieces you need and how to put them together. I picked an IMAP server as an example, because of the AHA! moment I had when I read the following in the in the Abe Fettig O'Reilly book" "To make an IMAP server, write classes to implement the IAccount, IMailbox, IMessage, and IMessagePart interfaces defined in twisted.mail.imap4" The next two sentences complete the description of the process, but the key point is that for most parts of Twisted, there isn't much that tells you how all of the various parts fit together. And Twisted is BIG. It's hard to find things sometimes... Some of this could be improved by expounding on the API docs for these interfaces, but I think a broad overview for many of these areas would help immensely.
More true in some places than others, but a definite problem in some places. A personal pet peeve of mine is the "What Deferreds don't do: make your code asynchronous" section of http://twistedmatrix.com/projects/core/documentation/howto/gendefer.html There's a nice big example of what Deferred's don't do. OK, great, but if I'm new to Twisted and the async style of coding in general, then what *DO* I do to make my code asynchronous? (NOTE: some of this I think will be helped in the upcoming Deferred documentation rewrite...see the "Deferred documentation rewrite" thread on this list for the last day or two)
Emphasis on "how it fits together", IMHO. Navigating Twisted's' documentation (outside the API docs) is a nightmare. I often find myself just having to use Google searches.
I looked at this a while back, but I found it pretty much incomprehensible.
Thanks!
Jean-Paul
A couple of other things: There are several barriers to improving Twisted's docs - you have to know a fair amount about Twisted to write about some parts of it - you have to know a lot about Twisted to write about Twisted from a high level, big-picture sort of perspective - you have to know some seriously deep Twisted magic to even speak intelligibly about certain things - once you know enough about Twisted to really explain it...it seems you are no longer able to explain it :) Not trying to offend...everyone in the Twisted community I've talked to has been very helpful, but sometimes when you know a complex topic really well, it becomes difficult to explain things in such a way that someone new to that topic will understand. Does this make sense? Sometimes I think what is needed is someone who knows basically nothing about Twisted to go about learning it...and then writing down how they did it. Also: Some (more) documentation about how to write docs for Twisted would really help people who are willing (and in some cases eager!) to contribute, but feel the barrier to entry is too high (they don't want to figure out pydoctor + epydoc + xml processing + ...). Anyway, I hope my rambling is helpful... Kevin Horn
![](https://secure.gravatar.com/avatar/53cf3ca941974336efe1921cdef8e83b.jpg?s=120&d=mm&r=g)
----- Original Message ----- From: Kevin Horn "Sometimes I think what is needed is someone who knows basically nothing about Twisted to go about learning it...and then writing down how they did it." I'm trying to do this, by keeping notes as I learn twisted to build a custom email system. It's hard to admit how dumb I feel sometimes though. Frequently what happens is I get stuck when something doesn't work, and while searching through the docs I stumble upon something I already read earlier and thought I understood, and in re-reading it I get an insight or better understanding of what the doc means now that I can connect it to a specific problem and I have more experience. That's not usually how I use or experience documentation. Generally I use documentation to first find out what the tool is able to do, what it's supposed to be good for and good at, and then when I want to exploit one of those things I drill down to the specific instructions for it. That's hard to do as the docs are now, proably because the specific instructions depend so much on other specific instructions you may not know about yet. -Dave
![](https://secure.gravatar.com/avatar/ca0bbb8ddab6982c5536b7a07b85c1e6.jpg?s=120&d=mm&r=g)
Hi Jessica! I'm quite a newbie at twisted yet, and a newcomer to the project, so I do think that twisted documentation is one of it's more problematic areas!. My suggestion would be that more than *adding* documentation, I would suggest to first search-and-destroy deprecated documentation. For me, one of the most frustrating parts of using twisted was reading through the twisted core doc, trying to do things and failing, and then accessing #twisted and get people tell me thinks like 'oh, tap/tac/tml/mktap are no longer used', or things like that (I don't yet know which ones are supposed to be used and which don't). I understand that *rewriting* outdated doc is a major task, that's why I just suggest deleting it ;) (maybe in a new Revised Twiste Doc section of the web site or the like), then it probably would be clearer where to start and what needs most work. It could also be a good idea to try to make it more wiki-like at the beginning; I think there's quite a lot of people that would like to contribute with this, but having different documents/styles for different areas of the doc is not good, a wiki like doc might allow some people to work on giving a more cohesive look to the whole doc, and others to add the meat and bones, provide examples, etc, while allowing everyone to review it easily. Thinking about it, of course a doc on SVN would help in the same way ;). Truth is that I really don't know which are twisted best practices at the time (what I do works, but I don't know if it's the best way to do it), and checking outdated doc just confuses things even more... better no doc at all. I stop my suggestions here before someone yells me 'then do it!' :). -- Santiago
![](https://secure.gravatar.com/avatar/dd1243740a09f0676ef225404105cfc0.jpg?s=120&d=mm&r=g)
On Jul 31, 2009, at 10:19 AM, Santiago Aguiar wrote:
I'd like to second this, for at least the third or fourth time over the last several years ;-) To respond in advance to previous criticisms of the idea: * wiki syntax sucks * it doesn't have an acceptable form of version control * it's frustrating that it can't be easily bundled with a release tarball * the same info is on the mailing list, just use google to find it All 100% true. Nevertheless, I say wiki. It would provide the least complicated route for new users to document things they discover during the learning process, which is *exactly* the phase that is so hard for the more experienced users to properly explain. Hopefully over time the various technical writers who volunteer often could rewrite this content into the formal collection that is distributed with Twisted proper, but in the meantime, a community- edited wiki would also provide another window into the community for new developers. I also would recommend MediaWiki. Yes, less than perfect, not as good as this, that, or the other wiki. In the interest of keeping the bar for entry low, however, we would be harnessing a familiarity with MediaWiki provided by exposure to Wikipedia. Pragmatism, not idealism, and all that... -phil
![](https://secure.gravatar.com/avatar/fcdfff68a2c9b2d1d199e4626998c791.jpg?s=120&d=mm&r=g)
On Fri, Jul 31, 2009 at 9:34 AM, Phil Christensen <phil@bubblehouse.org>wrote:
Well, if there *were* to be a change in the documentation system used, I would push for moving to Sphinx, rather than a wiki. * it's rapidly becoming a "standard" for docs in the Python world * it has lots of neat features * it can be version controlled * multiple output formats (html, chm, Latex(and therefore PDF), etc.) * I just like it :) Just sayin' Kevin Horn
![](https://secure.gravatar.com/avatar/f4b759bb1fd0b5d37695b7de428ae9c7.jpg?s=120&d=mm&r=g)
Hi All,
Rather than change the documentation system entirely, who don't we just create a new resource for Twisted documentation - a Twisted Documentation Wiki. At the very least it can be a repository of various tips and tricks we happen to find, and maybe it can even be a staging ground for documentation that makes it into Twisted proper. Reza -- Reza Lotun +44 (0)7521 310 763 rlotun@gmail.com
![](https://secure.gravatar.com/avatar/ca0bbb8ddab6982c5536b7a07b85c1e6.jpg?s=120&d=mm&r=g)
Reza Lotun wrote:
Even if I re-suggested the wiki based documentation, I think it's important to be extra careful on how it's used. One thing I personally hate is projects whose documentation is basically wiki-based, and what you end having is a disconnected set of tips, many out of date, of how to do this and that. It could be OK it it's labeled 'Tipi-wiki' but not 'Documentation' :). As you say, I think that the wiki could be useful as a staging ground to build a formal documentation, but it shouldn't be even suggested for new users, since what they will probably find are many unfinished ideas. Alternatively, a separate doc repo with sphinx based doc could be built so that it will allow for collaborative development making clear that is a work in progress and a product with 'releases'. I say a different repo to avoid having to give commit access to code for people who are working on doc, maybe same repo with different permissions would be better. And I suggest sphinx to a) start from scratch and add existing doc as we see it's relevant, b) have a more flexible base than HTML docs, and for the reasons mentioned by Kevin (plus I want to learn it :P). -- Santiago.
![](https://secure.gravatar.com/avatar/f4b759bb1fd0b5d37695b7de428ae9c7.jpg?s=120&d=mm&r=g)
I agree - the wiki shouldn't *replace* the documentation, but the reality is I have loads of bookmarks of blog posts and discussions on the mailing list, and it'd be nice if I could to go one place to find all that type of info. A "recipe" or "cookbook" wiki might be all we need, with the ability to comment on each. The Activestate Python Cookbook is kinda what I'm thinking about: http://code.activestate.com/recipes/langs/python/
I haven't yet looked at lore or sphinx, but is there a way to support sphinx via lore? Reza -- Reza Lotun +44 (0)7521 310 763 rlotun@gmail.com
![](https://secure.gravatar.com/avatar/dd1243740a09f0676ef225404105cfc0.jpg?s=120&d=mm&r=g)
On Jul 31, 2009, at 12:32 PM, Reza Lotun wrote:
I agree with all of the above. A wiki is nice when there is no suitable formal documentation available for a topic. I think I may have used the aphorism once before, "documentation is like sex, when it's good, it's great, when it's bad it's still better than nothing." (no offense intended to any with delicate sensibilities, btw ;-) My only question about Sphinx, isn't it just for API docs? Also, can it interpret Zope interfaces like pydoctor can? Personally I'm pretty happy with the API docs (although there's always room for improvement in the actual docstrings), I think if there's a documentation need that's more dire, it's the long-form instructional kind. I just don't want to sidetrack *that* discussion by getting into API documentation concerns. -phil
![](https://secure.gravatar.com/avatar/fcdfff68a2c9b2d1d199e4626998c791.jpg?s=120&d=mm&r=g)
On Fri, Jul 31, 2009 at 11:52 AM, Phil Christensen <phil@bubblehouse.org>wrote:
My only question about Sphinx, isn't it just for API docs? Also, can it interpret Zope interfaces like pydoctor can?
Sphinx is great for writing all sorts of docs. At its core, it's basically a system for gluing together a bunch of RestructiredText documents and automatically cross-linking them, building an index, a javascript-based search capability, handling tables of contents, and outputing the results in various formats. But then it's extensible...so you can write plugins for it. One of the earliest (and included with the base package) is the "autodoc" extension which makes generating nice clean API docs from your docstrings. What it does _not_ do is process all those epydoc bits that are in Twisted's docstrings, though you might be able to write an extension for that. It expects docstrings to be in RestructuredText. AFAIK Sphinx does not have support for Zope interfaces, but I haven't looked in a while, and I seem to remember that someone was working on adding it...though I may be making that up. At any rate, I think using Sphinx for Twisted's API docs (at this juncture, anyway) would be a bad idea. But Sphinx excels at the kind of "long-form, instructional" docs you mention below. I use it for all my PHP projects (which of course, the autodoc extension doesn't work on), and I'm really happy with it.
Agreed.
I just don't want to sidetrack *that* discussion by getting into API documentation concerns.
Double Agreed.
-phil
Kevin Horn
![](https://secure.gravatar.com/avatar/fcdfff68a2c9b2d1d199e4626998c791.jpg?s=120&d=mm&r=g)
On Fri, Jul 31, 2009 at 12:11 PM, Kevin Horn <kevin.horn@gmail.com> wrote:
OK, having Googled around, it seems I was *not* making this up. There is some support for Zope interfaces in Sphinx, though it looks pretty minimal at the moment. Just, you know, in case anyone was interested. :) Kevin Horn
![](https://secure.gravatar.com/avatar/2326f5e54bb073d5252a23b39708b839.jpg?s=120&d=mm&r=g)
A lot has been said in this discussion but I'll add one thing. On 07/31/2009 06:32 PM, Reza Lotun wrote:
What I miss is a comment system related to the API docs so that it is possible for beginners and others to add small notes there without having to make a ticket, wait for it to be assigned etc. Apart from that I will just welcome anyone who writes documentation. Kind regards, Tarjei
-- Tarjei Huse Mobil: 920 63 413
![](https://secure.gravatar.com/avatar/95658668bebdc20db10bb5ccc1603017.jpg?s=120&d=mm&r=g)
2009/8/3 tarjei <tarjei@nu.no>:
I have an old, half finished idea in this area, which is that pydoctor has a mode which runs a web server and allows you to edit the docstrings online. It needs a bunch of work to get to useful -- probably user accounts, a way of not losing all changes when you restart the server, a lot of polish and, indeed, maybe a comment feature or discussion view -- but I think the idea is sound. If you have pydoctor installed, you can try pydoctor --auto in a directory containing some python modules. Cheers, mwh
![](https://secure.gravatar.com/avatar/fcdfff68a2c9b2d1d199e4626998c791.jpg?s=120&d=mm&r=g)
On Fri, Jul 31, 2009 at 11:14 AM, Santiago Aguiar <santiago.aguiar@gmail.com
wrote:
In my experience, this what happens to every project that a) uses a wiki for core documentation and b) has more than one person editing the wiki.
I'd love to see a documentation "reboot" using Sphinx, but not if it's going to be a half-baked, never-finished project. Since Jessica started this thread, and is the only person (so far) who has given any sort of commitment to actually working on new/revised docs, I'd really like to hear her opinion. Jessica? You still out there? Hope we haven't scared you off... I'd also be interested in hearing the opinions of some of the core Twisted guys on the various things we've been talking about here. What do you guys think about using a different docs system than what is being used now? If you guys are all dead set against it, there's not much point hashing out the details... Also, what do the Twisted core devs think about having a secondary wiki/cookbook thingy outside of the core docs? Kevin Horn
![](https://secure.gravatar.com/avatar/7ed9784cbb1ba1ef75454034b3a8e6a1.jpg?s=120&d=mm&r=g)
On Fri, 31 Jul 2009 11:55:08 -0500, Kevin Horn <kevin.horn@gmail.com> wrote:
I don't find that people trying to use Twisted complain about the presentation of the documentation. I find that they complain about its content. So, I think that it's the content that needs to be addressed. I don't *think* that switching to Sphinx (or anything else) is going to make any different to the content of the current documentation. One might argue that Lore is a significant barrier to entry for new contributions to the documentation, but you'd have to try pretty hard to convince me. Pretty much anyone can write some simple (Lore x)html. And if they can't, then there are other people willing to translate plain text into Lore input documents.
Also, what do the Twisted core devs think about having a secondary wiki/cookbook thingy outside of the core docs?
As a staging area for development of future core docs, I think I would recommend using a version control system (perhaps a distributed one), not a wiki. As something intended to be user-facing, I don't think it's a great idea. Of course, there *is* a wiki hosted on the website already... And it has some documentation on it... So what's being proposed, exactly? :) Obviously I'm speaking for myself, not for all the other people who have contributed to Twisted. Jean-Paul
![](https://secure.gravatar.com/avatar/dd1243740a09f0676ef225404105cfc0.jpg?s=120&d=mm&r=g)
On Jul 31, 2009, at 1:11 PM, Jean-Paul Calderone wrote:
What I was suggesting originally was that a wiki would be a staging ground for those core docs. IMO, traditional version control is not generally well-suited for collaborative document development. I guess what I'm thinking is, no one likes writing docs, so when we have to good fortune to encounter someone who does, let's make it as easy as possible.
True. I guess there's only two considerations. 1. Using the current Trac installation for a user-edited wiki might convey that the contents are endorsed by Twisted proper. 2. Although I've become fairly used to it, Trac's wiki isn't the greatest, and more importantly, Trac itself sometimes seems to be held together with bubble gum and scotch tape. -phil
![](https://secure.gravatar.com/avatar/fcdfff68a2c9b2d1d199e4626998c791.jpg?s=120&d=mm&r=g)
On Fri, Jul 31, 2009 at 12:11 PM, Jean-Paul Calderone <exarkun@divmod.com>wrote:
True enough. And in my case, at least, navigation.
I don't think using Lore is a problem particularly...but it need to be documented more clearly, and all in one place. Right now, there's stuff on the tracwiki, in the main docs, on the mailing list, etc. Particularly, the answers to the questions: - How do I build the documentation using Lore? - What is the Lore xhtml syntax, and how should I use it? - What is the process for making a documentation contribution? Please note, that I was originally suggesting Sphinx primarily as an alternative to a wiki-based system, rather than as a replacement for Lore...but the conversation kind of mutated. :)
Agreed, wiki = yuck (for this). Even as a "staging ground". IMO trac-wiki is really only suited for "marketing" type content.
![](https://secure.gravatar.com/avatar/dd1243740a09f0676ef225404105cfc0.jpg?s=120&d=mm&r=g)
On Jul 31, 2009, at 1:34 PM, Kevin Horn wrote:
Okay, tell me again what exactly the problem is with a wiki? I feel like the same thing happens every time we discuss documentation. Someone makes a recommendation to do it the easy way, and someone else dismisses any solution that doesn't satisfy their programmer's OCD. Yes, I'd love to see version controlled XML documentation that adheres to a common writing style that is enforced across the board, but this talk has been happening for *YEARS* and there has been little improvement of significance (I have to emphasize that I understand a number of people have worked very hard on this, and I don't mean to denigrate their contributions). People keep telling me wikis are bad, but I'm still not getting the 'why' -- I just hear "wikis are bad for documentation" repeatedly presented as a fact. They do seem to work reasonably well for scores of other projects. PRAGMATISM!!! ;-) -phil
![](https://secure.gravatar.com/avatar/fcdfff68a2c9b2d1d199e4626998c791.jpg?s=120&d=mm&r=g)
On Fri, Jul 31, 2009 at 12:57 PM, Phil Christensen <phil@bubblehouse.org>wrote:
Well, pragmatism is my basic reason for not liking wikis for docs. :) For me, it boils down to every time I've worked on or with a project that used wikis for docs (assuming that the project is of at least moderate size and has more than a couple of editors) that documentation has been terrible. It's ended up as a disjointed mess, that's hard to navigate, and has a bajillion half-completed thoughts, and lots of outdated cruft. Of course, some could say the same of (at least parts of) the current Twisted documentation. ;) It's not that there's anything wrong with wiki's per se, it's just that they encourage "bad habits". If there were a solid editorial process in place, where someone was specifically responsible for reviewing, editing, splitting, merging, and correcting documents, then a wiki could probably work. But I think it's easier to get coherent docs using tools that encourage "good habits". Obviously my definitions of "good habits" and "bad habits" are pretty vague...and not entirely spelled out even in my own mind. Er, sorry about that... Kevin Horn
![](https://secure.gravatar.com/avatar/dd1243740a09f0676ef225404105cfc0.jpg?s=120&d=mm&r=g)
On Jul 31, 2009, at 2:15 PM, Kevin Horn wrote:
Indeed; I wouldn't use those particular words because of their pejorative qualities, but I think the end result is that way because of the level of interest in maintaining documentation for a particular project. That's a different issue, but it is the reason why I'd like the documentation process to be easy for 'regular developers' (e.g., not the proto-deities that make up the core devs ;-) to contribute to.
Well, honestly, I don't disagree with you at all. A good set of documentation requires a great deal of work, and the amount of rewriting and fact-checking required is proportional to the amount of development going on. Twisted development has slowed somewhat over the years in a few areas, but it still moves quite quickly in others. I don't see any documentation system making this go away. It will always have to be edited and revised, and it will often be out of date. I think the issue is one of not 'scaring away' potential writers with a cumbersome process. Correct me if I'm wrong, but the current process for writing Twisted documentation is as follows (at least for people without commit access): 1. check out a directory of Lore XHTML files 2. make my changes or create a new file, making sure not to work against the sparsely documented schema ;-) 3. generate a patch 4. create a ticket, attach the patch 5. bug someone on #twisted or the mailing list to do a review 6. wait some unspecified amount of time 7. code is merged into trunk 8. wait for the live docs to be regenerated Compared to a wiki: 1. create an account (let's assume it's email-verified) 2. write or edit some docs 3. see your docs live immediately Which do you think is going to be a more compelling process for people to get involved with? The first process is straight from the software development playbook, so it seems like a good idea, but I don't agree with the need for the same regimented process to exist w.r.t. documentation, especially when we are in such dire straights in that regard. I'm not really married to the idea of a wiki per se, it's just that it seems to be the closest thing that provides contributors a low bar for entry and instant gratification by seeing their work online (and on a list of recent changes). Plus, it addresses a concern Ying Li just posted as I was writing this: you automatically get revisions, last- modified dates, and diffs of those revisions. Yes, some of this documentation will not be of the quality we would like it to be. But it will still exist, which is a hell of a lot better than nothing. Plus, it gives the technical writers we occasionally see interest from some source material to work with in the first place, which has got to be helpful towards the ultimate goal of formally written core documentation. -phil
![](https://secure.gravatar.com/avatar/a7267ede9d6ea0117c5438c398a1a212.jpg?s=120&d=mm&r=g)
I am not scared off. In fact, I'm updating the wiki/ContributingToTwistedLabs and related how-to-edit-documentation pages right now. I think adding content to and restructuring the howtos gives me a full plate already, and to be honest I'm more interested in the content than the presentation. People who have more experience with the successes and failures of the various ways to organize project content online can make that decision and I'd happily help move content over to another format should that be necessary. Thanks for all the help and feedback, everyone. -Jessica On Fri, Jul 31, 2009 at 12:55 PM, Kevin Horn <kevin.horn@gmail.com> wrote:
![](https://secure.gravatar.com/avatar/fcdfff68a2c9b2d1d199e4626998c791.jpg?s=120&d=mm&r=g)
On Fri, Jul 31, 2009 at 12:16 PM, Jessica McKellar < jessica.mckellar@gmail.com> wrote:
Great! I think that this is a critical first step.
Fantastic. Content is definitely more important, and as this thread proves, its easy to get bogged down in the "how" rather than the "what".
Kevin Horn
![](https://secure.gravatar.com/avatar/e1554622707bedd9202884900430b838.jpg?s=120&d=mm&r=g)
On Fri, Jul 31, 2009 at 1:16 PM, Jessica McKellar < jessica.mckellar@gmail.com> wrote:
I see the edits in the timeline. They look great! Thanks! :-D Do you have any ideas for improving the front page as well? I think the random-list-of-links approach to its information architecture hasn't been terribly effective. I think adding content to and restructuring the howtos gives me a full plate
already, and to be honest I'm more interested in the content than the presentation.
This is good to hear. I think it should definitely be our first priority. Thanks for all the help and feedback, everyone.
Please don't hesitate to ask for any other help or feedback you might need. People who have more experience with the successes and failures of the
various ways to organize project content online can make that decision (...)
I don't want to stifle the possible discussion of using a different system, but for those of you interested in switching to Sphinx or something else, there are several things you need to address: First and foremost, you need to make a commitment to *maintaining* the documentation infrastructure. This includes teaching other folks how to use it. Second, you need to submit a complete proposal. This means we need a clear enumeration of what benefits the different documentation technology will have; this needs to be written up in one place so that we can evaluate it and discuss it without chasing down a million little email threads or ticket comments. Such a proposal also needs to address integration of automated testing and automated doc-building with the examples. You'll need to set up a builder that will replace our current documentation builder. You'll need to update all the relevant wiki pages. Third, you need to address the transition plan. Just translating all the documentation by hand is a possibility, but it has potential problems. What happens if we discover a problem with the new tool? How do we roll back? and I'd happily help move content over to another format should that be
necessary.
Lucky for you, Jessica appears to have volunteered to do this for you :-). (You might want to confirm that first, though.) I certainly wouldn't mind switching to a tool that has lots of fancy features that lore lacks, but a hit-and-run approach where we just switch tools in the hopes that it will make something better may leave us in a worse situation than we already are.
![](https://secure.gravatar.com/avatar/46299857e4644e581677e2b80811b0c4.jpg?s=120&d=mm&r=g)
On Jul 31, 2009, at 4:23 PM, Glyph Lefkowitz wrote: ...
I'm wondering if there's an inventory of the various types of documentation systems currently in use. Not just "systems" but mostly "stray documentation." From this discussion, it doesn't seem that the Lore formatted and built docs are as much of a problem as the fragmentation into tracwiki, main docs, mailing list, etc. that we've discovered as we've been discussing it. Since "everything" is not in the build system anyway, perhaps starting a branch, in a new build system (Sphinx), where we pull things in, one chunk at a time, will not be a hit and run approach, but will force the reorganization of the docs into one, actually unified format. Presentation is another issue, for another day. Then we can all see what's documented, what's not, what's old and out of date, etc. With a dating system as discussed elsewhere, the docs will actually improve over time instead of becoming more fractured and out of date with no traceability. Just my 13.4 cents. S
![](https://secure.gravatar.com/avatar/e1554622707bedd9202884900430b838.jpg?s=120&d=mm&r=g)
On Fri, Jul 31, 2009 at 11:10 PM, Steve Steiner (listsin) < listsin@integrateddevcorp.com> wrote:
"This time for sure, Rocky!" If you actually want to volunteer to do all of the work for this (which I outlined in my previous message) then feel free. But attempts to fix the world by blowing it up are rarely successful. A better approach would be to incrementally enumerate the things which have been covered on the mailing list and wiki, then pull them into the lore docs one at a time. Of course, the people who have actually volunteered their time, rather than their suggestions, seem to agree with this general outline :). Something you have to keep in mind with grand efforts like "let's rewrite all the docs in the hottest new ReFrumpledText format" is that Twisted is a product mostly of people's spare time, and therefore the person working on it may suddenly become busy and lose interest. When they *do* lose interest — possibly for a year or longer — we need to make sure that things are in a good state in the meanwhile, and their efforts have improved things. Each small improvement to the lore documentation will improve the overall documentation situation. As you put it, a big problem here is: the fragmentation into tracwiki, main docs, mailing list, etc
if the fragmentation is instead into tracwiki, main docs, sphinx docs, mailing list, etc, the problem has actually gotten *worse*, not better. Especially if the sphinx person walks away halfway through the effort, and then some other person comes along and says, "oh hey, what we should *really * do is rewrite all the docs in YAML", then rewrites another small subset of them and leaves. I hope you can see why I want to hold on to our current toolchain until we have someone around who has demonstrated a much deeper commitment to documentation than anyone yet has. For example, Steve, if you close 100 existing documentation tickets in the next week, then make the exact same suggestion again, I'll be a lot less resistant ;-).
![](https://secure.gravatar.com/avatar/e1554622707bedd9202884900430b838.jpg?s=120&d=mm&r=g)
On Sat, Aug 1, 2009 at 12:11 AM, Steve Steiner < listsin@integrateddevcorp.com> wrote:
I do tend to use a lot of words to get things across. Sorry ;-).
Where would we put an "inventory" project for it to be most useful?
As JP mentioned earlier in this thread, some work has already taken place here: <http://twistedmatrix.com/trac/wiki/DocumentationAnalysis>. If you'd like to contribute to that, please let somebody on the #twisted IRC channel know your trac username so we can set you up with wiki-edit permission.
![](https://secure.gravatar.com/avatar/46299857e4644e581677e2b80811b0c4.jpg?s=120&d=mm&r=g)
I get it, really.
I do tend to use a lot of words to get things across. Sorry ;-).
Duly noted ;-).
It doesn't really look like that effort went anywhere. On: http://twistedmatrix.com/trac/wiki/DocumentationAnalysis/DocumentList The first document is User reviewed by JoeBlow with Expert Reviews by JaneBlogs. There are only five more entries and only one is "completed" with a User and Expert review. What the heck that properly vetted doc has to do with anything is not particularly clear. Maybe we should have an auction? Auction: Document blah-de-blah Supplier will document blah-de-blah, work with Sponsors to incorporate blah-de-blah into their apps, and release the copyright to the resultant documentation and copyright to the Twisted Software Foundation. Supplier will provide up to X hour of consulting to each of the first three Sponsors (Founding Sponsors) to help refine the documentation and make sure it can be used in real-world projects. Minimum Founder bid is $50. Minimum Contributor bid is $35. Sponsor1: I'll give $50 for someone to explain blah-de-blah and how to use it in my app Sponsor2: I'd give another $50 if I could get this working in my app Supplier1: I'd have to have $200 to take this on. I'll also offer discounted consulting on the same topic at $??/hr. to anyone who contributes $35 or more to this documentation project after Sponsor3: I'll chip in the extra $100 if I can be one of the first three. I'm really stuck on blah-de-blah and could use some help. Moderator: Ok, we seem to have a deal. Sponsor 1 is in for $50, Sponsor 2 is in for $50, Sponsor 3 is in for $100. Each of them receives X hours of help from Supplier 1 which will be used to advance their projects and improve the documentation. Any future Contributors, over $35, receive discounted consulting from Supplier1 at $??/hr. for any work related to this issue as well as access to beta versions of the documentation and early-release versions of the sample code. Please sign off on this agreement and submit your payment. Supplier1 agrees to provide a first draft within X-period-of-time. Consulting arrangements begin then. The Twisted Software Foundation accepts a donation of 10% of all transactions for facilitating these transactions and to generally advance Twisted. (http://twistedmatrix.com/trac/wiki/TwistedSoftwareFoundation ) Just a thought... S P.S. I apologize in advance for any formatting issues with the above. Obviously, I'm not following the coding spec and pylint barfed on the whole thing... ;-). s
![](https://secure.gravatar.com/avatar/e1554622707bedd9202884900430b838.jpg?s=120&d=mm&r=g)
On Sat, Aug 1, 2009 at 11:30 PM, Steve Steiner (listsin) < listsin@integrateddevcorp.com> wrote:
I don't think it's quite fair to say it didn't go *anywhere*, it just didn't get anywhere particularly complete. I think the reviews that have been done thus far are still a valuable resource for someone interested in working on the documentation.
Maybe we should have an auction?
(snipped lengthy description of labor-intensive bounty resolution process) What we really need is for people in the community to knuckle down and do the work of actually reviewing the documents, rewriting things, soliciting feedback, copy-editing the documents, checking for accuracy, etc. As Jessica did when she started the thread :). Frankly, the work is boring: what the documentation needs is not a dynamic market-driven system for managing contributions and paying bounties, it's not a comprehensive overhaul of our entire documentation toolchain. What it needs is for people to sit down and read all the documentation, make a list of all of its problems, one at a time, and then write, delete, and edit as necessary, incorporating feedback from the community. Once we have people taking part in that effort and working with the docs on a daily or weekly basis, *then* we can listen to the reality of the problems and frustrations that they have working with the tools or determining priorities or whatever and solve those problems. Separately from that, if you're interested in contributing some time to manage a bounty program for the TSF, that might be a reasonable idea. Other projects have experimented with bounty-based systems for resolving bugs. Unfortunately there are a number of difficulties with those systems, which are really out of the scope of this discussion, but they're worth talking about if you have a real interest in doing it.
![](https://secure.gravatar.com/avatar/46299857e4644e581677e2b80811b0c4.jpg?s=120&d=mm&r=g)
On Aug 2, 2009, at 12:41 AM, Glyph Lefkowitz wrote:
Ok, fine, it went "over there." Now...if you're just starting out, how much does "over there" help and how much has "over there" ever been integrated into anything else? How would you even find "over there?" I guess I'm still frustrated with the lack of an overarching doc organization scheme. I've tried to use various parts of the Twisted eco-system, on a number of projects, and have repeatedly gotten stuck on outdated, disjointed, disorganized, and just plain wrong documentation. As I've said before, the complete lack of date/version stamping on example code has been the greatest source of frustration. I've, more than once, gotten half-way through a tutorial and realize that it just plain didn't work with the current version. End of exploration. I used the best result I found in my search and it just didn't work. Gotta move on; client's getting annoyed. Unbillable time; bad.
Hasn't happened, and isn't going to happen, in my opinion, until there's some actual incentive for "people in the community", which is a small subset of the "potential community", to actually do anything about it. Documentation is not needed by the "community" as much as it's needed by the "potential community."
Frankly, the work is boring and it ain't happening, never has, and never will unless something changes. Bounty system, torture, Target coupons, whatever...
See above. How many years does the same thing have to happen for there to be a realization that, without a fundamental change, nothing is going to happen? If you paid someone, 20 hours a week, to babysit the documentation effort, they would. Depending on their competence, there would be progress. If you wait, endlessly for "Once we have people taking part in that effort and working with the docs on a daily or weekly basis", you're going to get the same result.
I'm not sure what I'm interested in, at this point, but I certainly am not interested in doing it for free and neither is anyone else. In my opinion, and by existing evidence, coherent documentation ain't happening, and won't, unless something fundamental changes. It's not my project and it's not for me to decide but it seems to me that waiting for "people to take part in that effort" is doomed to waiting forever... S
![](https://secure.gravatar.com/avatar/307ebad5fc7824b9f223fbad5698d278.jpg?s=120&d=mm&r=g)
On Fri, Jul 31, 2009 at 10:19 AM, Santiago Aguiar<santiago.aguiar@gmail.com> wrote:
Something that I find irritating about a lot of documentation that I read is that it is not dated. All the twisted docs are labeled with a version (currently 8.2.0) but this is meaningless to me. I think a big improvement would be putting in a small bit of context around the documentation , such as: * when the howto/tutorial was last updated * what version (of deferreds/imap/whatever the howto/tutorial is about) the howto/tutorial was written for What would be even more useful, although it would also be a lot more work, is to perhaps link to a diff between the code as it is now and as it was when the tutorial was written. Or just a changelog. Also, I oppose deleting outdated documentation outright, as I think limited information is better than none. However, I think it should be clearly marked as such, so readers understand that the doc may not give them correct instructions. (This does not apply to the API docs as since the API docs are automatically generated, they should always current given the Twisted coding standards.)
![](https://secure.gravatar.com/avatar/46299857e4644e581677e2b80811b0c4.jpg?s=120&d=mm&r=g)
On Jul 31, 2009, at 3:37 PM, Ying Li wrote:
That is one of my pet peeves about project documents of any type. Tutorials, and examples especially. There's nothing more frustrating than getting 1/2 way through a tutorial, only to realize that the rest of it is just not going to work with the current version and that all that effort was wasted because that's not how it's done any more. If it's not dated, and perhaps tagged with the first version with which it was released, and the most recent version at which it was modified, it's impossible to figure out what's current, relevent, and likely to actually work with the current release. S
![](https://secure.gravatar.com/avatar/46299857e4644e581677e2b80811b0c4.jpg?s=120&d=mm&r=g)
On Jul 31, 2009, at 10:25 PM, Itamar Shtull-Trauring wrote:
Sorry for not being familiar with the overall system, but where would one put a suggestion regarding the overall date/time/version stamping of every document in the system? I'd be happy to propose a standard format but I'm not sure where to put it so it's universally applied (across all products). I know Jessica McKellar <jessica.mckellar@gmail.com> started working on the "ContributingToTwistedLabs" wiki how-to-edit-documentation docs. Can we get it in there, or where would be the best place to get this started? I've CC's Jessica to see how she thinks this might be integrated as well. Thanks, S AKA/ssteiner AKA/ssteinerX@gmail.com
![](https://secure.gravatar.com/avatar/307ebad5fc7824b9f223fbad5698d278.jpg?s=120&d=mm&r=g)
On Fri, Jul 31, 2009 at 10:25 PM, Itamar Shtull-Trauring<itamar@itamarst.org> wrote:
Could you file some bugs to that effect, so we don't forget these suggestions? Thanks!
Done: http://twistedmatrix.com/trac/ticket/3945 The formatting on the description is off though (I tried to make bullet points and failed), and I don't have permission to change it.
participants (15)
-
Dave Britton
-
Drew Smathers
-
Glyph Lefkowitz
-
Itamar Shtull-Trauring
-
Jean-Paul Calderone
-
Jessica McKellar
-
Kevin Horn
-
Michael Hudson
-
Phil Christensen
-
Reza Lotun
-
Santiago Aguiar
-
Steve Steiner
-
Steve Steiner (listsin)
-
tarjei
-
Ying Li